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HOW IMPORTANT IS YOUR DATA? 


Years of family photos. Your entire music 
and movie collection. Office documents 
you've put hours of work into. Backups for 
every computer you own. We ask again, how 
important is your data? 


NOW IMAGINE LOSING IT ALL 


Losing one bit - that’s all it takes. One single bit, and 
your file is gone. 


The worst part? You won't know until you | 
absolutely need that file again. Example of one-bit corruption 


THE SOLUTION 


The Mini boasts these state-of-the- 


The FreeNAS Mini has emerged as the clear choice to 
art features: 


Save your digital life. No other NAS in its class offers 


i ry and ZFS bitr 
ECC (error correcting code) memory and ZFS bitrot ee ee ee 


protection to ensure data always reaches disk . Up to 16TB of storage capacity 
without corruption and never degrades over time. - 16GB of ECC memory (with the option to upgrade 
to 32GB) 


« 2x 1 Gigabit network controllers 
No other NAS combines the inherent data integrity : Ramotemansuementoort (EN 


and security of the ZFS filesystem with fast on-disk - Tool-less design; hot swappable drive trays 
encryption. No other NAS provides comparable power oRiSe NPS ictal emanecomngurey 

and flexibility. The FreeNAS Mini is, hands-down, the 
best home and small office storage appliance you can 
buy on the market. When it comes to saving your 
important data, there simply is no other solution. 
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FREENAS 


CERTIFIED 
STORAGE 


With over six million downloads, 
FreeNAS is undisputedly the most 
popular storage operating system 
in the world. 


Sure, you could build your own FreeNAS system: 
research every hardware option, order all the 

parts, wait for everything to ship and arrive, vent at 
customer service because it hasn't, and finally build it 
yourself while hoping everything fits - only to install 
the software and discover that the system you spent 
days agonizing over isn’t even compatible. Or... 


MAKE IT EASY ON YOURSELF 


As the sponsors and lead developers of the FreeNAS 
project, ixsystems has combined over 20 years of 
hardware experience with our FreeNAS expertise to 
bring you FreeNAS Certified Storage. We make it 
easy to enjoy all the benefits of FreeNAS without 
the headache of building, setting up, configuring, 
and supporting it yourself. As one of the leaders in 
the storage industry, you know that you're getting the 
best combination of hardware designed for optimal 
performance with FreeNAS. 


Every FreeNAS server we ship is... 


» Custom built and optimized for your use case 

» Installed, configured, tested, and guaranteed to work out 
of the box 

» Supported by the Silicon Valley team that designed and 
built it 

» Backed by a 3 years parts and labor limited warranty 
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As one of the leaders in the storage industry, you 
know that you're getting the best combination 

of hardware designed for optimal performance 

with FreeNAS. Contact us today for a FREE Risk 
Elimination Consultation with one of our FreeNAS 
experts. Remember, every purchase directly supports 
the FreeNAS project so we can continue adding 
features and improvements to the software for years 
to come. And really - why would you buy a FreeNAS 
server from anyone else? 


FreeNAS 1U 

- Intel® Xeon® Processor E3-1200v2 Family 

- Up to 16TB of storage capacity 

- 16GB ECC memory (upgradable to 32GB) 

« 2x 10/100/1000 Gigabit Ethernet controllers 
- Redundant power supply 


FreeNAS 2U 
- 2x Intel® Xeon® Processors E5-2600v2 Family 
- Up to 48TB of storage capacity 
- 32GB ECC memory (upgradable to 128GB) 
« 4x 1GbE Network interface (Onboard) - 
(Upgradable to 2 x 10 Gigabit Interface) 
« Redundant Power Supply 


http://www.iXsystems.com/storage/freenas-certified-storage/ 
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EDITORS’ WORD 


Dear Readers, 


A S spring Is here, it is high time to wish you that Easter will bring 
you a lot of joy, happiness, hope and light. 


Our wishes for this Easter. 
Good health, 
Good fortune, 
Fulfilling life. 


| hope that this issue of BSD magazine will be a good lecture 
for the holidays. | know that all of you are waiting for holidays 
so | do not bother you more. Please go to the Table of Contents 
page to see what we prepared and what you will find inside this 
BSD issue. Just start reading now. 

Finally, | would like to thank you Authors, Reviewers, Proofreaders, 
BSD fans, Friends, and Readers for your invaluable support and 
contribution. 


Happy Easter! 
Ewa & BSD Team 
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IN BUSINESS 


FreeNAS 
in an Enterprise Environment 


By the time you're reading this, FreeNAS has been downloaded 
more than 5.5 million times. For home users, it’s become an 
indispensable part of their daily lives, akin to the DVR. 
Meanwhile, all over the world, thousands of businesses 
universities, and government departments use FreeNAS to 
build effective storage solutions in myriad applications 
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* How TrueNAsS meets modern storage challenges for entery 
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FreeBSD World 
C Developer in a FreeBSD World. Part 2 S 


David Carlier 

In the “The Journey of a C developer in a FreeBSD World’, 
David described the changes that occur when you land in a BSD 
system coming from Linux. Now, you get ready to get your C/ 
C++ code working in both platforms; this time we will look into 
the debugging side. 


Expert says ... 


A Complete Guide to FreeNAS Hardware 
Design, Part Il: Hardware Specifics 

Josh Paetzel 

A guide to selecting and building FreeNAS hardware, written 
by the FreeNAS Team, is long past overdue by now. For that, 
we apologize. The issue was the depth and complexity of the 
subject, as you'll see by the extensive nature of this four part 
guide, due to the variety of ways FreeNAS can be utilized. 
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security 
Does your information belong 
to the CIA triad? 14 


Rob Somerville 

Confidentiality, Integrity and Availability are the three pillars 
of Information Security. In this article, we pose a number of 
scenarios to you the IT professional and ask What would you 
do? Every environment is different, so we will not provide any 
answers. Instead we want to stimulate thought and debate 
around the ethics that Donn Parker says are missing from the 
computer center. 
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Industrial Network Security Securing Critical 
Infrastructure Networks for Smart Grid, 
SCADA, and Other Industrial Control 
Systems 

Eric D. Knapp & Joel Thomas Langill 

The first step of information analysis requires a certain degree of 
data collection so that there is a healthy body of data to assess. 
Collecting evidence relevant to cyber security requires knowing 
what to monitor and how to monitor it. You will learn about 
determining what to monitor, successfully monitoring security 
zones, information management and, log storage and retention. 
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Column 


With a former intelligence operative confirming 
that the NSA has developed the prized technique 
of concealing spyware in the firmware of hard 
drives, what are the implications and is there 
any point in shutting the door now that the 


horse has bolted? 42 
Rob Somerville 
Interview 
Interview with with Solene Rapenne 44 
Luca Ferrari 
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Download syslog-ng Premium Edition 
product evaluation here 


Attend to a free logging tech webinar here 


BalaBit 


IT Security 


www.balabit.com 


syslog-ng log server 


The world's first High-Speed Reliable Logging™ technology 


HIGH-SPEED RELIABLE LOGGING 


m above 500 000 messages per second 


m zero message loss due to the 
Reliable Log Transfer Protocol™ 


= trusted log transfer and storage 


The High-Speed Reli 
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C Developer ina 
FreeBSD World. Part 2 


In the “The Journey of a C developer in a FreeBSD World’, | 
described the changes that occur when you land in a BSD 
system coming from Linux. Now you, dear readers, get ready 
to get your C/C++ code working in both platforms; this time 
we will look into the debugging side. Indeed, FreeBSD’s 

libc has jemalloc builtin. OpenBSD contains its specific 
implementation, called ottomalloc. 


s a C/C++ developer, you have concerns about 
memory leakage, corrupted memory. In the pre- 


vious issue, the article “GDB debugger” perfectly 


described its proper usage. Its reading is greatly recom- 
mended. We'll focus on the memory allocators. 

In OpenBSD, several options are available via the MAL- 
LOC_OPTIONS environment variable or the global mal- 
loc_options variable changeable from within your C/C++ 
code. To enable a specific option, it is the uppercase let- 
ter. To disable it, it is the lowercase counterpart. 

The malloc statistics are disabled by default, for perfor- 
mance matters. In order to enable these, the OpenBSD 
source code is needed and we just need to uncomment 
this line IN 1ib/libc/stdlib/malloc.c: 


/* #define MALLOC STATS */ 


Then we can just recompile the libc: 


> od lib/libe 


> make obj && make depend && make install 


Now, malloc_dump symbol is available! 
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> nm /usr/lib/libc.a | grep malloc dump 
00000790 T malloc dump 


One | find quite useful is the junk option (enabled by de- 
fault since 5.6 release). 

Indeed, after an allocation, the memory area is filled 
with Oxd0O. When it is freed, it is filled with Oxdf. What you 
can spot easily is when you try to use a previously freed 
memory pointer. 


int 

main(int argc, char *argv[]) 

{ 
char *p = new char[4]; 
strlcpy (py “too’, -4)7 
ScLae1COut. << p>. << Sstdisendl; 
delete p; 
Std: cout: << p <<. srdizendL,; 


return (0); 
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You ought to see this kind of output: 


> ./test 
foo 
XXXXXXXXXXXX 


=> we have our filled free pointer ... 


> env MALLOC OPTIONS=j ./test => let’s disable the junk 
option 

foo 

foo 

=> not good at all, we can believe the p pointer is still 


valid at this point ... 


As you can see above, the junk option is really useful 
and the performance hit is quite acceptable so it is quite 
advised to keep this option on, even in production. 

The Freeguard option, F, is useful for detecting double free. 


Ihc 
main(int argc, char *argv[]) 


{ 


char *p = malloc(sizeof(char) * count); 


free (p); 
<i.e no realloc meanwhile ...> 


free (p);7 


return (0); 


> ./test 
=> The double free not caught ... 


> env MALLOC OPTIONS=F ./test 

test(7086) in free(): error: bogus pointer (double free?) 
0x7820972f£40 

Abort trap (core dumped) 

=> The double free is caught 


Another flag useful for debugging, A (for abort, enabled 
by default) which simply coredumps the current process. 

Previously, we compiled the libc to enable the malloc 
Statistics, hence to enable D (for Dump) statistics. So 
compiled with debug symbols: 
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int 
main(int argc, char *argv[]) 
{ 
char *p = malloc(4096); 
malloc dump (2); 


return (0)? 


> ./test 

=> from within the code, malloc dump prints on the given 
file descriptor those statistics ... 

Malloc dir of test at 0xlld6c987f3d0 

Region slots free 511/512 

Finds 0/0 

Inserts 1/0 

Deletes 0/0 

Cheap reallocs 0/0 

Free chunk structs: 

Free pages cached: 0 

slot) hash d type page f 
size [free/n] 

65) # 65 0 pages 0x11d6a9f84000 Oxlldéaa3ed86d 4096 

In use 16384 

Guarded 0 

Leak report 


ig sum it avg 


0x11d6aa3ed86d 4096 1 4096 
Here we got the faulty address, 0xlidéaa3ea86a, when 


the pointer was allocated but never freed. 
Or we can call malloc_dump from within gdb 


> gdb ./test 

After putting a breakpoint to exit, we call call malloc_ 
dump to print on stderr 

malloc dump (2) 


Oxfa0110012704096 1 4096 


list *0xf£a011001270 

We retrieve our faulty code here: 
int 

main(int argc, char *argv[]) 


{ 
char *p = malloc(4096); 
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malloc dump (2); 


return (0); 


Apart of MALLOC OPTIONS, OpenBSD protects well 
against stack overflow’s issues. The stack protector flag 
is implied; there’s no need to add it for the compilation. 


int 
main(int argc, char *argv[]) 
{ 
char buf[4]; 
Sstrepy (burt, “Loobar”’); 
pence ( sen", but), 


> ec -g -OZ -o test test.c 
/tmp/ccPGztFB.o(.text+0Oxldb): In function ‘main’: 
: warning: strcpy() is almost misused, please use 


str bepys ) 


> ./test 
foobar 


Abort trap (core dumped) 


When possible, it is advised to use stricpy (the compiler 
is nice enough to warn you about that), except if you're 
100% confident about the source you attempt to copy. 


int 

Maln(iAt “Arde, char *argv ||) 

{ 
char buf [4]; 
setrlopy (but, “toobar’, 7); 
pornce(™se no"; but); 


> CC =—G =O02 =o Test. test.c 

test.c: In function ‘int main()’: 

test.c:9 warning: array size (4) smaller than bound length (7) 
=> you have been warned that your buffer is really too 


small ... 


For FreeBSD, it is slightly different but we can retrieve 
similar options with jemalloc, like junk. 

As a developer, you might need to make sure that MAL- 
LOC PRODUCTION is not defined in either /etc/src. 
conf and /etc/make.conf. Although it brings significant 
performance improvements, the debugging capabilities 
are lost. 
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Tie 

main(int argc, char *argv[]) 

{ 
char *p = new char[4]; 
strlepy (py “too’, 4)4 
std::cout << p << std: endl; 
delete p; 
Std: scout << p << srdizendl; 


return (0); 


> ./test 
foo 


LLZLLZLZLLZ4Z4 


> setenv MALLOC CONF “junk:false” 
> ./test 
foo 


foo 
It is possible to dump statistics via stats_ print options: 


> setenv MALLOC CONF “stats print:true” 
> ./test 


__ Begin jJemalloc statistics 

Version: 3.6.0-0-g46c0af68bd248b04df£75e4£92d5fb804c3d75340 

Assertions enabled 

Run-time option settings: 
opt.abort: true 
opt.tg chunk? 22 
opt.dss: “secondary” 
opt.narenas: 32 
Opt.l¢g dirty multy 2 
Optwetats Print: true 
opt.junk: true 
opt.quarantine: 0 
opt.redzone: false 
opt.zero: false 
opt.utrace: false 
opt.xmalloc: false 
opt.tcache: true 
Opt. 9G. teaches max: 15 

CPUS: 2 

Arenas: 32 

Pointer size: 8 

Quantum size: 16 


Page size: 4096 
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Min active:dirty page ratio per arena: 8:1 printr (“ss\n"); 
Maximum thread-cached size class: 32768 } 
Chunk size: 4194304 (2%22) 
Allocated: 4096, active: 4096, mapped: 8388608 meu 
Current active ceiling: 4194304 Main(int argc, char *argv[]) 
chunks: nchunks highchunks curchunks { 
2 2 2 char *p = malloc (4096); 
huge: nmalloc ndalloc allocated /* could simply make the first argument as NULL */ 
0 0 0 malloc stats: print (im stats; NUL, NULL) ; 


return (0); 
arenas[0]: } 
assigned threads: 1 
dss allocation precedence: secondary With the utrace option, it adds an entry for ktrace... 


dirty pages: 1:0 active:dirty, 0 sweeps, 0 madvises, 0 


purged > setenv MALhOC CONF “utrace;true” 
allocated nmalloc ndalloc nrequests > ktrace ./test 
small: 0 0 0 0 > kdump 
large: 4096 il 0 1 
total: 4096 1 0 1 1245 test CALL utrace(0x/7fffffffeaa0, 0x18) 
active: 4096 1245 test USER 0x8010060000 = malloc (4096) 
mapped: 4194304 1245 test RET utrace 0 
bins: bin size regs pgs allocated nmalloc 1245 test CALL utrace(0x/ffftffffeaa8, 0x18) 
ndalloc nrequests nfills nflushes 1245 test USER free(0x801006000) 
newruns reruns curruns 
[O.a2e] 
large: size pages nmalloc ndalloc nrequests curruns 
4096 1 1 0 1 1 As you can see without any third party tool, in FreeBSD 
[1017] / OpenBSD we have those features which greatly help 
--- End jemalloc statistics --- with debugging. Even if the price to pay is having low- 
er performance results, it is worth it during the develop- 
or from within the code: ment at least. 


David Carlier has been a developer since 2001, has been using BSD 


void since 2004 and has worked for a mobile based position as a C/C++ 

m stats(void *args unused, const char *data) developer in Ireland since 2012. During his spare time, he contributes 

{ to various BSD projects, especially FreeBSD, and writes some articles 
if (data != NULL) for BSDMag. 
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Hardware Design, 


Part Il: Hardware Specifics 


General Hardware Recommendations 
I've built a lot of ZFS storage hardware and have two decades of experience with FreeBSD. 
The following are some thoughts on hardware. 


Intel Versus AMD 
FreeNAS is based on FreeBSD. FreeBSD has a long history of working better on Intel than AMD. 
Things like (but not limited to) the watchdog controllers, USB controllers, and temperature moni- 
toring all have a better chance of being well supported when they are on an Intel platform. This is 
not to say that AMD platforms won’t work, that there aren’t AMD platforms that work flawlessly 
with FreeNAS, or even that there aren't Intel platforms that are poor choices for FreeNAS, but all 
things being equal, you'll have better luck with Intel than AMD. 

The Intel Avoton platforms are spendy but attractive: ECC support, low power, AES-NI support 
(a huge boon for encrypted pools). On the desktop side of things, there are Core i3 platforms with 
ECC support, and of course there are many options in the server arena. The single socket E3 
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EXPERT SAYS... 


Xeons are popular in the community, and of course for higher end systems, the dual 
package Xeon platforms are well supported. 


® 


Storage Controllers 

LSI is the best game in town for add-on storage controllers. Avoid their MegaRAID solutions and 
stick with their HBAs. You'll see three generations of HBAs commonly available today. The old- 
est (and slowest) are the SAS 2008 based I/O controllers such as the 9211 or the very popular 
IBM M1015. The next generation of these controllers was based on the 2308 which added PCI 
3.0 support and increased CPU horsepower on the controller itself. An example here is the 9207. 
Both the 2008 and 2308 based solutions are 6Gbps SAS parts. The newest generation of con- 
trollers are 12Gbps parts such as the 9300. The FreeNAS driver for the 6 Gbps parts is based on 
version 16 of the stock LSI driver with many enhancements that LSI never incorporated into their 
driver. In addition, many of the changes after version 16 were specifically targeted at the Inte- 
grated RAID functionality that can be flashed onto these cards. As a result, “upgrading” the driver 
manually to the newer versions found on the LSI website can actually result in downgrading its 
reliability or performance. | highly recommend running version 16 firmware on these cards. It’s the 
configuration tested by LSI, and it’s the configuration tested by the FreeNAS developers. Running 
newer firmware should work, however running older firmware is not recommended or supported 
as there are known flaws that can occur by running the FreeNAS driver against a controller with 
an older firmware. FreeNAS will warn you if the firmware on an HBA is incompatible with the driv- 
er. Heed this warning or data loss can occur. The newer 12Gbps parts use version 5 of the LSI 
driver. Cards using this driver should use version 5 of the firmware. 

Most motherboards have some number of SATA ports built in. There are certain models of Mar- 
vell and J-Micron controllers that are used on motherboards that have large numbers of SATA 
ports. Some of these controllers have various compatibility issues with FreeNAS, and some of 
these controllers also have forms of RAID on them. As a general rule, the integrated chipset AHCI 
SATA ports have no issues when used with FreeNAS, they just tend to be limited to 10 ports (and 
often far fewer) on most motherboards. 


Hard Drives 

Desktop drives should be avoided whenever possible. In a desktop, if an I/O fails, all is lost. For 
this reason, desktop drives will retry I/Os endlessly. In a storage device, you want redundancy at 
the storage level. If an individual drive fails an I/O, ZFS will retry the I/O on a different drive. The 
faster that happens, the faster the array will be able to cope with hardware faults. For larger ar- 
rays, desktop drives (yes, I’ve seen attempts to built 1PB arrays with ZFS and desktop drives) 
are simply not usable in many cases. For small to medium size arrays, a number of manufactur- 
ers produce a “NAS” hard drive that is rated for arrays of modest size (typically 6-8 drives or so). 
These drives are worth the additional cost. 

At the high end, if you are building an array with SAS controllers and expanders, consider get- 
ting the nearline 7200 RPM SAS drives. These drives are a very small premium over Enterprise 
SATA drives. However, running SATA drives in SAS expanders —while supported— is a less de- 
sirable configuration than using SAS end to end due to the difficulty of translating SATA errors 
across the SAS bus. 


iXsystems Director of IT 
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SECURITY 


Does your information 
belong to the CIA triad? 


Confidentiality, Integrity and Availability are the three pillars 
of Information Security. In this article, we pose a number of 
scenarios to you the IT professional and ask What would you 
do? Every environment is different, so we will not provide 
any answers. Instead we want to stimulate thought and 
debate around the ethics that Donn Parker says are missing 


from the computer center. 


Question 1. 

A senior manager has a vital dead- 
line for early Monday morning. As 
part of this deadline, they must com- 
pose a very dense presentation of 
images, video and music from me- 
dia legally stored and appropriate- 
ly licensed on the corporate server over the weekend at 
home. This request arrives late on a Friday evening, and 
due to the size of the media, the only available hard-drive 
is an external USB drive that contains data confidential 
to the organisation. Transfer of the data via other means 
is impossible due to the total file sizes. The manager in 
question is renown for losing or breaking items. As there 
is not sufficient time to securely wipe the drive, a standard 
disk format is applied, in the knowledge that the confiden- 
tial information could be recovered fairly easily by a com- 
petent professional. Should IT inform the manager of this 
fact and ask them to be extra vigilant? 
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Question 2. 

In addition to the scenario in Ques- 
tion 1, the venue where the public 
presentation will take place does not 
have a Performing Rights Society li- 
cence (or global equivalent) to play 
the background music to the pre- 
sentation. IT are aware of this issue, but past experience 
has shown that advising managers about these facts are 
inevitably met with resistance, censure and in some cases 
verbal abuse. Whose duty is it to inform the manager? 


Question 3. 

A laptop is returned by a member of 
staff for a major upgrade. Should an 
audit be performed on the data down- 
loaded from the Internet and websites 
visited as a matter of course? Internet 
provision is supplied free of charge 
by the organisation to staff members working from home. 
Where are the lines drawn between corporate and personal 
use? If illegal content was found (e.g. pirated music or vid- 
eos) who would hold legal liability if a) Access was via a cor- 
porate VPN / Firewall or b) Access was direct to the Internet? 
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Question 4. 
With the discovery of spyware now 
being embedded in the firmware 
of hard drives', what action can IT 
take to remedy this attack vector? 
Is there a policy in place to inform 
senior management of the risks? 
If the response of management 
to this risk is non-committal or worse still, deriso- 
ry, how can IT protect itself from the buck being 
passed down the line in the case of an incident? 


Question 5. 

Manager (A) demands ad- 
ministrator rights to a system 
and you refuse, offering an ad- 
equate alternative. You have 
verbal evidence from Manager 
(A) that they wanted this access 
for unethical reasons. Your manag- 
er (B) is the best friend of Manager (A). How would you 
address this scenario without compromising Manager (B) 
or yourself? 


Question 6. 

As a result, sometime later in your 
annual review you are marked down 
by Manager (B) for being uncooper- 
ative. You also discover that Manag- 
er (B) is quite at ease with Manag- 
er (A)’s behaviour, despite the risks 
to the organisation. With hindsight, how would this affect 
your response to Question 5? 


Question 7. 

You discover a major security flaw in 
a public facing Internet system and 
alert your manager. Both you and 
your manager agree that this sys- 
tem is not fit for purpose and should 
be recommissioned. Senior man- 
agement overrides both you and your manager and the 
flaw remains un-patched, due to cost. What do you do? 


Question 8. 

Consequently, the flaw is exploited 
and a customer reports the breach. 
The system is decommissioned, but 
you are warned by your employ- 
er not to discuss this with anyone. 
The system is widely used; do you 


1 http://www.reuters.com/article/2015/02/16/us-usa-cyberspying-id USKBNOLK1QV20150216 
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inform the IT departments in other organisations as the 
vendor is attempting to cover up the issue and will not fix 
it in a speedy manner? 


Question 9. 

A member of staff (A) has gone on 
sick leave and another member of 
staff in the same department (B) 
wants access to their data and email 
for business purposes. Approval for 
this should come from their respec- 
tive manager (C), but he cannot be contacted to give ap- 
proval, and you deny the request. The member of staff (B) 
reports this to senior management and you are told to give 
(B) access by senior manager (D). Consequently, (A) and 
manager (C) make an official complaint against you as 
(B) has accidentally sent confidential information to a third 
party using (A)’s email account. (D) washes their hands of 
the whole affair and has support of senior management 
who would prefer to lay blame at the door of IT. How do 
you proceed? 
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Question 10. 

Your manager (A) refuses to give 
(B) access to an external VPN of 
a partner organisation as it is a 
known security risk. You witness the 
exchange, and while your manag- 
er’s response could be considered 


brusque, it is not aggressive or threatening. Shortly after, 
you find (B) in tears and report this to your manager (A). 
(B) attempts to make a formal complaint against (A). If 
the complaint is investigated (A)’s manager, (C) will lead 
the process. You are approached by an influential senior 
member of staff (D) and are effectively told if you are ap- 
proached as a witness to lie about the incident. You can- 
not tell (A) and (D) is frequently seen with (C) who is re- 
portedly ‘on-side’ with (D). How do you proceed if a) you 
are asked to be a witness and b) you have no faith in 
the formal whistle-blowing policy? All the others you know 
who have followed this path have been dismissed or have 
resigned under duress. 


Question 11. 

Consequently, no formal complaint is 
raised. Do you make a formal com- 
plaint against (D) to your manager 
(A) knowing full well that (A) has an 
axe to grind with (D) and that (C) will 
probably take the side of (D)? 


Question 12. 
| All Internet and Email traffic is moni- 


tored and logged in your organisa- 
tion. A personal witch-hunt is being 
performed against a popular, pro- 
fessional and effective member of 
staff. Do you provide the logs ‘as is’ 
knowing that any minor infringements of company poli- 
cy (e.g. Facebook use, sending personal emails etc.) — 
which are normally ignored — will be used against them? 


Question 13. 

The company responsible for facilities 
management has access to all areas 
of the building, but you suspect that 
they are tampering with equipment 
in the datacenter. Furthermore, they 
have allowed 3rd parties unsuper- 


vised into the datacenter without notifying IT on numerous 
occasions. Access is via key-card, but senior management 
refuse to allow you to install an additional key-lock as the 
FM company is responsible for the infrastructure and they 
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demand access. Apart from generating a bureaucratic “Told 
you so” audit trail, what can you do to remedy matters? 


Question 14. 

The CEO of the organisation has 
asked you to securely delete certain 
key original documents. You know 
that the media and national press 
are anxious to obtain the originals. 
Multiple historical backups exist, 
but the individual file cannot be removed from these 
without destroying the backups. How rigorously do you 
carry out your instructions, if at all? Would it be ethical 
to release the document to the press in direct defiance 
of instructions? 


Question 15. 

Manager (A) who was instrumental 
in head-hunting you for your senior 
role is being investigated for stealing 
intellectual property from a compa- 
ny you both worked for some years 
ago. While you were not party to this 
event, you always believed manager (A)’s account that 
there was a major disagreement and everything was set- 
tled so when you are questioned you defend the manager 
as you have no evidence to the contrary. Some months lat- 
er, a customer demands a brand-new PC for his daughter 
otherwise he will withhold substantial payment. You raise 
your concerns with (A), but are told to deliver a new com- 
puter despite your protests. Some months later, manager 
(A) (along with others) resign due to major conflicts with 
senior management. You then discover major financial im- 
proprieties have been taking place throughout the organ- 
isation and you are being pressurised to fraudulently sign 
compliance certification which are legally binding docu- 
ments. You refuse, and are ostracised by the senior part- 
ners. Do you resign on principle? 


Rob Somerville has been passionate about technology since his ear- 
ly teens. A keen advocate of open systems since the mid-eighties, he 
has worked in many corporate sectors including finance, automo- 
tive, airlines, government and media in a variety of roles from tech- 
nical support, system administrator, developer, systems integrator 
and IT manager. He has moved on from CP/M and nixie tubes but 
keeps a soldering iron handy just in case. 
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Securing Critical Infrastructure Networks for Smart Grid, 
SCADA, and Other Industrial Control Systems 


Security Monitoring of Industrial Control Systems 


The first step of information analysis requires a certain 
degree of data collection so that there is a healthy body of 
data to assess. Collecting evidence relevant to cyber security 
requires knowing what to monitor and how to monitor it. 


could be relevant to cyber security, and because 
there are many unknown threats and exploita- 
tions, even information that may not seem relevant today 
may be relevant tomorrow as new threats are discovered. 
Even more unfortunate is that the amount of seemingly 
relevant data is already overwhelming — sometimes con- 
sisting of millions or even billions of events in a single day, 
with even higher rates of events occurring during a period 
of actual cyber-attack.' It is therefore necessary to assess 
which events, assets, applications, users, and behaviors 
should be monitored — as well as any additional relevant 
systems that can be used to add context to the informa- 
tion collected, such as threat databases, user information, 
and vulnerability assessment results. 
An additional challenge arises from the segregated na- 
ture of a properly secured industrial network. Deploying 
a single monitoring and information management sys- 


| nfortunately, there is a lot of information that 


1 J.M. Butler. Benchmarking Security Information Event Management (SIEM). The SANS Institute Analytics 
Program, February, 2009. 
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tem across multiple otherwise-separated zones violates 
the security goals of those zones and introduces poten- 
tial risk. The methods used to monitor established zones 
must be considerate of the separation of those zones, and 
the data generated from this monitoring need to be man- 
aged accordingly as well. While there are benefits to fully 
centralized information management, the information be- 
ing generated may be sensitive and may require “need 
to know” exposure to security analysts. Therefore, cen- 
tralized monitoring and management needs to be overlaid 
with appropriate security controls and countermeasures, 
up to and including full separation — forgoing the efficien- 
cies of central management so that the analysis, informa- 
tion management, and reporting of sensitive information 
remains local in order to maintain absolute separation of 
duties between, for example, a highly critical safety sys- 
tem and a less secure supervisory system. 

In order to deal with massive volumes of log and event 
data that can result from monitoring established network 
zones, and the challenges of highly distributed and seg- 
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regated zones, best practices in information management 
— including shortand longterm information storage — must 
be followed. This is necessary in order to facilitate the 
threat detection process, and also as a mandate for rel- 
evant compliance requirements, such as the North Ameri- 
can Electric Reliability Corporation Critical Infrastructure 
Protection (NERC CIP), NRC Title 10 CFR 73.54, Chemi- 
cal Facility Anti-Terrorism Standards (CFATS), and others 
(see Chapter 13, “Standards and Regulations’). 


DETERMININGWHAT TO MONITOR 

The trite answer to “what to monitor” is “everything and 
more!” Everything that we monitor, however, results in in- 
formation that must be managed. Every data point results 
in a log record, or perhaps a security or safety alert. As- 
sets, users, applications, and the communication chan- 
nels that interconnect them all require monitoring. Be- 
cause there are so many assets, users, applications, and 
networks that need to be monitored, the total amount of 
information generated every second in even a moderately 
sized enterprise can be staggering.? While products exist 
to automate security event and information management, 
the total amount of information available can quickly over- 
whelm the information analysis and storage capacity of 
these tools. Therefore, security monitoring requires some 
planning and preparation in order to ensure that all nec- 
essary information is obtained, without overloading and 
potentially crippling the tools the information is intended 
to feed. 

One approach is to segregate monitoring by zone. Just 
as the separation of functional groups into zones helps 
minimize risk, it also helps to minimize the total informa- 
tion load that is generated by that zone. In other words, 
there are limited assets and activities within a zone, and 
therefore there are less total logs and events. 

To further complicate matters, operational technology 
(OT) activities and metrics must also be considered when 
securing industrial networks — representing new data types 
from yet another potentially overwhelming source of new 
assets such as remote terminal units (RTUs), programma- 
ble logic controllers (PLCs), intelligent electronic devices 
(IEDs), and other industrial assets; applications such as 
human—machine interfaces (HMIs), and Historians; and 
networks such as fieldbus and smart grid networks. 


TIP 

When considering network monitoring and information 
management, it is helpful to benchmark the information 
load currently being produced in both IT and OT net- 
works. IT networks require identifying which devices need 


2 Ibid. 
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to be monitored. This means understanding what servers, 
workstations, firewalls, routers, proxies, and so on (almost 
every IT device is capable of producing logs of some sort) 
are important — the process of determining critical assets 
described in Chapter 2, “About Industrial Networks,” and 
Chapter 9, “Establishing Zones and Conduits,” is helpful 
here. Once it has been determined which devices need to 
be monitored, the event load generated by these devices 
needs to be calculated. One method is to measure the 
event load of a period of time that contains both normal 
and peak activity, and divide the total number of events 
by the time period (in seconds) to determine the average 
event per second (EPS) load of the network. Alternately, 
a worst-case calculation can be based entirely on peak 
event rates, which will result in a higher EPS target.° 

Most assets in OT networks, mainly the embedded de- 
vice types, like PLCs, RTUs, and IEDs, which make up 
the majority of network-attacked assets, do not produce 
events or logs at all, and therefore they cannot be mea- 
sured. However, they do produce information. This can be 
easily derived by looking at historized data from the con- 
trol plants, and/or through the use of specialized indus- 
trial protocol monitors. Determine which assets you wish 
to monitor, and use the Data Historian system to deter- 
mine the amount of information collected from these as- 
sets over time. This information will need to be normalized 
and centralized — either automatically via an SIEM or simi- 
lar product, or manually via human time and effort — so it 
may be prudent to limit the amount of historized data that 
need to be exposed for security assessment. Some His- 
torian tags — especially system tags concerning authenti- 
cation, critical alarm tags concerning point or operational 
changes, stopped or failed processes, and so on — are 
obvious choices, while others may have little relevance 
to security. This step is effectively a form of security event 
“rationalization,” similar to the process performed on the 
process event systems of ICS to improve operational ef- 
fectiveness. 

Once the initial benchmark is obtained, add room for 
growth, and room for headroom — perhaps 10% (this will 
vary by situation). When sizing the IT network, it is also 
prudent to plan for “peak averages” where peak traffic rates 
occur for extended periods of time (i.e. the peak becomes 
the average), as this condition can occur during an extend- 
ed attack, or as a result of a successful breach and subse- 
quent infection with malware.* Unusual peak averages may 
also occur on OT systems during abnormal events, such as 
plant startups and shutdowns, or during system patching or 
on-process migrations and upgrades. OT systems may re- 


3 Ibid. 
4 Ibid. 
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port different conditions but are less likely to report higher 
numbers of conditions unless the control process being his- 
torized has been significantly altered. 

So what really needs to be monitored? The following guide- 
lines help to identify wnat systems should be monitored. 


SECURITY EVENTS 

Security events are those events generated by security 
and infrastructure products: networkor host-based fire- 
walls, network routers and switches, malware prevention 
systems, intrusion detection and prevention systems, ap- 
plication monitors, and so on. Ideally, any event generat- 
ed by a security device should be relevant, and therefore, 
these devices should be used for promiscuous monitor- 
ing. Realistically, false positives can dilute the relevance 
of valid security events. 


NOTE 

The term “false positive” is often misused. False positives 
are often associated with what are seemingly irrelevant 
security data because security logs and events originate 
from many sources and are often generated quickly and 
in large quantities. When an alert is generated because 
a benign activity matches a detection signature of an in- 
trusion detection system (IDS), the result is a false posi- 
tive. Similarly, if an anti-virus system falsely indicates that 
a file is infected, the result is a false positive. False posi- 
tives make security analysis more difficult by generating 
extra data points that need to be assessed, potentially 
clouding real incidents from detection. 

False positives can be minimized through tuning of the 
faulty detection signatures — a process that should be per- 
formed regularly to ensure that detection devices are op- 
erating as efficiently as possible. While false positives of- 
ten result in large amounts of unnecessary or irrelevant 
data, not all irrelevant data are false positives. Many se- 
curity analysts and even security vendors are tempted to 
overly tune devices to eliminate any alert that occurs in 
large numbers because of this common misconception. 
The issue with overly aggressive tuning is that while it 
will make incidents easier to manage in day-to-day op- 
erations, it can introduce false negatives — that is, when 
a real threat fails to create an alert, or when a correla- 
tion rule fails to trigger because a necessary condition 
was suppressed by over-tuning (see Chapter 11, “Excep- 
tion, Anomaly, and Threat Detection”). Remembering that 
event correlation signatures are signature-matching rules 
that detect known threat patterns, the elimination of small- 
er seemingly irrelevant events can prevent detection of 
the larger pattern. Similarly, as security researchers dis- 
cover new patterns, event data that seem irrelevant today 
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may become relevant in the future (See Figure 1). To en- 
sure accurate threat detection and correlation, all legiti- 
mately produced events should be retained short-term for 
live analysis (i.e. kept on-line) and long-term for forensic 
and compliance purposes (i.e. kept off-line) regardless of 
how irrelevant they may seem at the time of collection. 
Only true false positives — the events generated due to a 
false signature match — should be eliminated via tuning 
or filtering. 

When considering the relevance of security events in 
industrial networks, consider the source of the event and 
its relevance to the specific zone being monitored. For ex- 
ample, all zones should have at least one perimeter secu- 
rity device, such as a firewall or IPS, but there may also 
be multiple host-based security devices capable of gen- 
erating events, such as anti-virus, application whitelisting, 
intrusion detection and prevention systems (HIDS/HIPS), 
firewalls, or other security devices (see Chapter 9, “Estab- 
lishing Zones and Conduits”). One example is industrial 
security appliances 
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That use industrial protocol and application monitoring 
to enforce how industrial protocols are used. 

These logs might provide much more specific data to 
a zone than do general security events, as seen in the 
example below from a Tofino industrial security appliance 
that provides detailed information pertaining to the unau- 
thorized use of an industrial protocol (Modbus/TCP) func- 
tion code (6 = “write single register’): 


May20- 09: 20250 169.254.2 <2Apr 14 19247232 
V0 502C2Z = B3125 256 

CEF:1|TofinoSecurity Inc|TofinoSA|02.0.00|300008|TofinoModbus/ 

TCPEnforcer:Function Code List Check|6.0|msg = 
Functioncode 6 isnot in permitted function code list 
TofinoMode = OPERATIONAL smac = 9c:eb:02:a6:22 src = 
192.168.1.126 spt = 32500 
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dmac= 00:00:bc:cf:6b:08 dst = 192.168.1.17 dpt = 502 proto 
= TCP TofinoEthType = 800 TofinoTTL= 64 TofinoPhysIn = eth0 


In contrast, a generic Snort IDS might produce a syslog 
event string identifying a perimeter policy violation, such 
as the attempted Windows update shown below, but 
cannot provide the context of application function codes 
within the industrial network (See Chapter 6, “Industrial 
Network Protocols”). 

Jan 01 00:00:00 


[69220259259] snort: [12200294876] ETPOLICY 


External Windows Updatein Progress [**] [Classification: 
Potential Corporate Privacy Violation] [Priority:1] 
(TCP) ~L0.1 21023321665 
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An often-overlooked step prior to commissioning any 
device that will generate security events is to “tune” or 
validate that normal traffic does not trigger events. Fig- 
ure 2 illustrates how a complete rule set for a Tofino Se- 
curity Appliance might look once commissioned. Note 
that only the last rule (as indicated by the arrow) is ac- 
tually enforcing segregation on the conduit by perform- 
ing deep-packet inspection on Modbus/TCP (502/tcp) 
traffic originating in the ICS Host zone and destined for 
the ICS Controllers zone. There are many other types 
of valid traffic that is generated to support functionality 
like the Network Neighborhood used in Windows operat- 
ing systems and Neighboring Switches/Routers typical in 
both IT and OT network devices that is commonly sent 
to broadcast and multicast addresses. This valid traffic, if 
not properly handled with “drop-no log” entries in the rule 
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set would generate “false positives” in terms of the secu- 
rity events within an industrial network. Some of the traf- 
fic that must be considered include 


¢ Windows NetBIOS Traffic - Name Resolution Service 
(137/udp) and Datagram Server (138/udp) 

¢ Multicast DNS (5353/udp) 

e Link-Layer Multicast Name Resolution (5355/udp) 

e Universal Plug ‘n Play (1900/udp and 2869/tcp) 

¢ Web Services Discovery Protocol (3702/udp) 

¢ Cisco Discovery Protocol 

e Link Layer Discovery Protocol 

¢ Internet Control Message Protocol (IP Protocol 1) 

¢ Internet Group Management Protocol (IP Protocol 2) 

¢ Internet Protocol Version 6 (IPv6). 


ASSETS 
Assets — the physical devices connected to the network 
— also provide security data, typically in the form of logs. 
Assets can produce logs that track activity on a variety of 
levels. The operating system itself produces many logs, in- 
cluding system logs, application logs, and file system logs. 
System logs are useful for tracking the status of devices 
and the services that are (or are not) running, as well as 
when patches are (or are not) applied. Logs are useful for 
determining the general health of an asset, as well as vali- 
dating that approved ports and services are running. These 
logs are valuable in tracking which users (or applications) 
have authenticated to the asset, satisfying several compli- 
ance requirements. The following represents individual re- 
cords from a Redhat Linux system log showing a success- 
ful user login, and a Windows failed authentication: 


& Tofino - Firewall 
2 Rule Table 
“3 CThe firewall rules configured for this Tofino SA 
hom Bs ——— = : = rs 
nL | Asset Interface _| Direction | Asset Interface _| Protocol 
O lv] Any Net 1 = Any Net 2 fi, ARP 
vt ) Any Net 1 ir Any Net 2 (i, IPv6 
Fe wv] Local Net Net 1 or as Local Net Broadcast Net 2 4, NetBIOS-NS ~ vu 
3 | gg Local Net Net 1 Sr a Local Net Broadcast Net 2 #, NetBIOS-DS > 4 
= (4) gpg Local Net Net 1 ee 224.0.0.251 Net 2 4, mDNS oF 
on vl =. Local Net Net 1 = 224.0.0.252 Net 2 4), LLMNR = a 
2 S 7] ag Local Net Net 1 — iy Multicast - 224 Net 2 4, IGMP a) 
= ¢ | gay Local Net Net 1 my 239.255.255.250 Net 2 é), UPnP - UDP < ma 
® o ea ae Local Net Net 1 i 239.255.255.250 Net 2 @, WS-Discovery 13 
ee | gg Local Net Net 1 or gay Multicast - 239 Net 2 4}, IGMP au 
a 9 Hr t 
ew) ¥| as Local Net Net 1 = Any Net 2 4), ICMP 
same) eo MY Bis Hosts Net 1 =a ICS Controllers  Net2 @, MODBUS/TCP 


Figure 2. Tuning an industrial network security appliance 
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<345>Mar1l7 11:23:15 localhostsshd[27577]:Accepted password 
forknapp trom 2+fftfi10.1.1.1port. 2695. -ssnz 

<345>Fri Marl7 11:23:152011 680 Security SYSTEM User 
Failure 

Audit ENTERPRISEAccountLogon attempt by: MICROSOFT _ 
AUTHENTICATION PACKAGE V1 QLogonaccount: KNAPP Source 
Workstation: ENTERPRISEError Code:0xCQQ0006A 4574 


Although syslog is ubiquitously used across a variety of 
systems, other event logging systems are used as well 
— the most notable of which is the Windows Manage- 
ment Instrumentation (WMI) framework. WMI produces 
auditable events in a structured data format that can be 
used against scripts (for automation) as well as by other 
Windows operating system functions.’ Because syslog 
is SO widely supported, WMI events are often logged us- 
ing a Windows syslog agent, such as Snare for Windows 
to stream WMI events over syslog. It is also possible to 
configure log forwarding between Windows hosts when 
restrictions prohibit the installation of agents on critical 
assets using the Windows Event Collector functionality. 

The following WMI event example indicates the creation 
of a new process on a Windows server: 


Computer Name:WIN-OZ6H21NLQ05 

EventCode: 4688 

Type: Audit Success(4) UserName: 

Category: Process Creation Log File Name:Security 
Sceing | abi] so-l-5-19 

String[%2]: LOCAL SERVICE String[%3]: 
String[%4]:0x3e5 

51 20xc008 

6]:C:\Windows\System32\RacAgent.exe 

> $1936 

8] :Oxc5e4 


NT AUTHORITY 


role) 


String [ 


fo\\e) 


String | 


a\e 
~~ 


String [ 


ole 


String [ 
Message: Anewprocess has been created.Subject: Security 
ID; 
S-1-5-19Account Name: LOCAL SERVICEAccountDomain: 
NT AUTHORITY LogonID: 0x3e5 Process 
Information:NewProcessID: 0xc008 New ProcessName: C:\ 
Windows\System32\RacAgent.exe Token Elevation Type: 
TokenElevationTypeDefault (1)CreatorProcessID: Qxc5e4 
Token ElevationType indicatesthe type of token thatwas 
assigned to the newprocessinaccordance with User 
Account Control policy. Typelisa fulltoken with no 
privileges removed orgroups disabled.Afull token 
isonly used if User Account Controlis disabled or 
ifthe user is the built-in Administrator account 
or a serviceaccount. Type2isan elevated token with 
no privileges removed or groups disabled.Anelevated 


5 Microsoft. Windows Management Instrumentation. http://msdn.microsoft.com/en-us/library/ 
aa394582(v=VS.85).aspx, January 6, 2011 (cited: March 3, 2011). 
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token isused when User Account Controlisenabled 

and the user chooses to startthe program usingRunas 
administrator.Anelevated token isalso used when an 
application isconfigured toalways require administrative 
privilegeor to always require maximumprivilege, and 
theuseris a member of theAdministratorsgroup. Type 

3is a limited token withadministrative privileges 
removed andadministrativegroups disabled. The limited 
token isused when User Account Controlis enabled, the 
applicationdoes not requireadministrativeprivilege, and 
theuserdoes not choose to start theprogram using Runas 


administrator. 


The same event, when collected via syslog using a WMI 
agent, such as Snare, might look like this: 


<12345> Fri Marl7 11:23:15 2011]| |WIN- 
OZ6H2Z1INLQ05| | 4688] |Audit 

Success (4) ||||ProcessCreation||Security||S-1-5-19| | LOCAL 

SERVICE| |NT AUTHORITY | |0x3e5| |0xc008||C:\Windows\System32\ 


RacAgent. exe| |%%1936||Oxc5e4 


Application logs (covered in more detail under the sec- 
tion “Applications”) provide a record of application-spe- 
cific details, such as logon activities to an HMI, config- 
uration changes, and other details that indicate how an 
application is being used. These Application Logs are an 
important component in the security associated with ma- 
ny ICS applications since these applications common- 
ly utilize a single Windows logon authentication account 
and manage individual user actions via local application 
accounts and security settings. 

File system logs typically track when files are created, 
changed, or deleted, when access privileges or group own- 
erships are changed, and similar details. File system log- 
ging is included in Windows using the Windows File Pro- 
tection (WFP) within WMI, which is an “infrastructure for 
management data and operations on Windows- based 
operating systems.”* File monitoring in Unix and Linux 
systems is performed using auditd, as well as with other 
commercial file integrity monitoring (FIM) products, such 
as Tripwire (www.tripwire.com) and nCircle (www.ncircle. 
com). These logs are extremely valuable for assuring the 
integrity of important files stored on an asset — such as 
configuration files (ensuring that the asset’s configurations 
remain within policy), and the asset's log files themselves 
(ensuring that logged activities are valid and have not been 
tampered with to cover up indications of illicit behavior). 


6 Ibid. 
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CONFIGURATIONS 

Configuration monitoring refers to the process of monitor- 
ing baseline configurations for any indications of change, 
’ and is only a small part of Configuration Management 
(CM). Basic configuration monitoring can be done at a ru- 
dimentary level through a combination of host configura- 
tion file monitoring (to establish the baseline), system and 
application log monitoring (to look for change actions), 
and FIM (to ensure that configurations are not altered). 
While this does not provide true CM, it does provide an 
indication as to when established configurations are al- 
tered, providing a valuable security resource. 

Full CM systems provide additional key functions, typi- 
cally mapping at least partially to the security controls out- 
lined in NIST SP 800-53 under the section “Configuration 
Management,” which provides a total of nine configuration 
management controls:° 


¢ Configuration management policy and procedures — 
establishes a formal, documented configuration man- 
agement policy. 

¢ Baseline configurations — identifying and document- 
ing all aspects of an asset’s configurations to create 
a secure template against which all subsequent con- 
figurations are measured. 

¢ Change control — monitoring for changes and com- 
paring changes against the established baseline. 

¢ Security impact analysis — the assessment of chang- 
es to determine and test how they might impact the 
security of the asset. 

¢ Access restrictions for change — limiting configuration 
changes to a strict subset of administrative users. 

¢ Configuration settings — identification, monitoring, 
and control of security configuration settings and 
changes thereto. 

¢ Least functionality — the limitation of any baseline 
configuration to provide the least possible functional- 
ity to eliminate unnecessary ports and services. 

¢ Information service (IS) component (asset) inventory 
— establishing an asset inventory to identify all assets 
that are subject to CM controls, as well as to detect 
rogue or unknown devices that may not meet base- 
line configuration guidelines. 

¢ Establishment of a configuration management plan — as- 
signing roles and responsibilities around an established 
CM policy to ensure that CM requirements are upheld. 


Configuration management tools may also offer automat- 
ed controls to allow batch configurations of assets across 


7 National Institute of Standards and Technology, Special Publication 800-53 Revision 3. Recommended Security 
Controls for Federal Information Systems and Organizations, August, 2009. 
8 Ibid. 
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large networks, which is useful for ensuring that proper 
baselines are used in addition to improving desktop man- 
agement efficiencies. For the purposes of security moni- 
toring, it is the monitoring and assessment of the configu- 
ration files themselves that is a concern. This is because 
an attacker will often attempt to either escalate user priv- 
ileges in order to obtain higher levels of access, or alter 
the configurations of security devices in order to penetrate 
deeper into secured zones — both of which are detectable 
with appropriate CM controls in place. 

The logs produced by the CM are therefore a useful 
component of overall threat detection by using change 
events in combination with other activities, such as an 
event correlation system. For example, a port scan, fol- 
lowed by an injection attempt on a database, followed by 
a configuration change on the database server is indica- 
tive of a directed penetration attempt. Change logs are 
also highly beneficial (and in some cases mandatory) for 
compliance and regulatory purposes, with configuration 
and change management being a common requirement 
of most industrial security regulations (see Chapter 13, 
“Standards and Regulations’). 


TIP 

The problem with Configuration Management within ICS 
is that a large portion of the critical configuration informa- 
tion is retained in embedded devices often running pro- 
prietary or closed operating systems using nonstandard 
communication protocols. These devices (PLCs, RTUs, 
IEDs, SIS, etc.) represent the true endpoint with a con- 
nection to the physical process under control, making 
their configuration details (control logic, hardware con- 
figuration, firmware, etc.) one of the most critical compo- 
nents pertaining to the operational integrity of the ICS. 
While several available IT products, such as Tripwire, So- 
larwinds, and What’sUpGold, can provide configuration 
and change management for servers, workstations, and 
network devices, specialized products, such as Cyber In- 
tegrity™ by PAS and the Industrial Defender Automation 
Systems Manager from Lockheed Martin, provide not only 
the necessary database components to identify and track 
configuration changes, but an extensive library of system 
and device connectors necessary to extract configuration 
data from ICS components. 


APPLICATIONS 

Applications run on top of the operating system and per- 
form specific functions. While monitoring application logs 
can provide a record of the activities relevant to those 
functions, direct monitoring of applications using a dedi- 
cated application monitoring product or application con- 
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tent firewall will likely provide a greater gran- ularity of all 
application activities. Application logs can indicate when 
an application is executed or terminated, who logs into 
the application (when application-level security is imple- 
mented), and specific actions performed by users once 
logged in. The information contained in application logs 
is a summary, as it is in all log records. A sample appli- 
cation log record generated by an Apache web server is 
provided here: 


Jam “Ol 00200200 [69.20.32 012)93.80.237.221.— = 
[24/ Feb/2011:01:56:33 -0000] “GET/spambot/ 
spambotmostseendownload. 

php HTTP/1.0” 500 71224 “http://yandex.ru/yandsearch?text 
= video. 

krymtel.net” “Mozilla/4.0 (compatible; MSIE6.0; WindowsNT 
5.1; MRA 4.6 (build01425))” 


A corresponding application log entry from an ICS illus- 
trating a local access level change is shown here: 


Jan 01 00:00:00 ICSSERVERIHMI1LEVEL SecurityLevel Admin 
Jan 01 00:00:00 ICSSERVERIHMI1LEVEL Securityhevel Oper 


For a more detailed accounting of application activity, an 
application monitoring system can be used. For exam- 
ple, while it is possible that malware might be download- 
ed over HTTP, and be indicated in a log file, such as the 
first example shown earlier, monitoring an application's 
contents across a session could indicate malware that is 


embedded in a file being downloaded from an otherwise 
normal-seeming website, as shown in Figure 3. 


NETWORKS 
Network flows are records of network communications, 
from a source to one or more destinations. Network infra- 
structure devices, such as switches and routers, usually 
track flows. Flow collection is typically proprietary to the 
network device manufacturer (e.g. Cisco supports Net- 
Flow, and Juniper supports J-Flow), although many ven- 
dors also support the sFlow standard (see Table 1). 
Monitoring flows provides an overview of network usage 
over time (for trending analysis, capacity planning, etc.) 
as well as at any given time (for impact analysis, security 
assessment, etc.), and can be useful for a variety of func- 
tions, including? 


¢ Network diagnosis and fault management. 

¢ Network traffic management or congestion management. 

¢ Application management, including performance 
management, and application usage assessments. 

¢ Application and/or network usage accounting for bill- 
ing purposes. 

¢ Network security management, including the detec- 
tion of unauthorized devices, traffic, and so on. 


Network flow analysis is extremely useful for security 
analysis because it provides the information needed 
to trace the communications surrounding a_ security 
incident back to its source. For example, if an application 


9 Floworg. Traffic Monitoring using sFlow. http://www.sflow.org/sFlowOverview.pdf, 
2003 (cited: March 3, 2011). 


Figure 3. Application sessiondetails from an application monitor 
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Table 1. Network Flow Details 


SNMP interface indices (iflndex in IF-MIB) 


Flow start time 


Flow end time 


Number of bytes/ 
packets 


Source and destination 
IP addresses 


Source and destination port 


The size of the flow in terms of traffic 
volume (bytes, packets, etc.), as well as 
errors, latency, discards, physical addresses 
(MAC addresses), etc. 


When a network communication was 
initiated and when it ended 

Collectively, the start and 

stop timestamps also indicate the duration 
of a network communications 

Indicates the “size” of the network flow, 
indicative of how much data is being 
transmitted 


Indicates where a network communication 
began and where it was terminated 


Note that in non-IP industrial networks, the 
flow may terminate at the IP address of an 
Ml or PLC even 

though communications may continue over 
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SNMP details can provide indications of 
abnormal protocol operation that might 
indicate a threat 

More germane to industrial networks, 
the presence of interface errors, latency, 
etc. can be directly harmful to the correct 
operation 

of many industrial protocols (see Chapter 6, 
“Industrial Network Protocols”) 

Essential for the correlation of 
communications against security events 


Useful for the detection of abnormal 
network access, large file transfers, as 
might occur during information theft (e.g. 
retrieving a large database query result, 
downloading sensitive files, etc.) 

Essential for the correlation of related logs 
and security events (which often track IP 
address details) 

IP addresses may also be used to determine 
the physical switch or router interface of 
the asset, or 

even the geographic location of the asset 


specialized industrial network protocols 


whitelisting agent detects malware on an asset, it is 
extremely important to know where that malware came 
from, as it has already breached the perimeter defenses 
of the network and is now attempting to move laterally 
and infect adjacent machines. By correlating the malware 
attempt to network flows, it may be possible to trace the 
source of the malware and may also provide a path of 
propagation (i.e. where else did the virus propagate). 
Network flow analysis also provides an indication of net- 
work performance for industrial network security. This is im- 
portant because of the negative impact that network per- 
formance can have on process quality and efficiency, as 
shown in Table 1. An increase in latency can cause certain 
industrial protocols to fail, halting industrial processes.'° 


CAUTION 

It is important to verify with the ICS supplier that network 
flow functionality can be enabled on the industrial network 
without negatively impacting the performance and integ- 


10 B. Singer, Kenexis Security Corporation, in: D. Peterson (Ed.), Proceedings of the SCADA Security Scientific 
Symposium, 2: Correlating Risk Events and Process Trends to Improve Reliability, Digital Bond Press, 2010. 
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rity of the network and its connected devices. Many indus- 
trial protocols include real-time extensions (see Chapter 
6, “Industrial Network Protocols”) that see switch perfor- 
mance issues when available forwarding capacity has 
been altered. Network vendors like Cisco have addressed 
this with special “lite” capabilities for netflow reporting. Al- 
ways consult the ICS supplier before making modifica- 
tions to recommended or qualified network topologies and 
operating parameters. 


USERIDENTITIESAND AUTHENTICATION 

Monitoring users and their activities is an ideal method 
for obtaining a clear pic- ture of what is happening on the 
network, and who is responsible. User monitoring is al- 
so an important component of compliance management, 
as most compliance regulations require specific controls 
around user privileges, access credentials, roles, and be- 
haviors. This requirement is enforced more so on systems 
that must comply with requirements, such as 21 CFR 
Part 11 and similar standards common in “FDA- regulated 
industries,” such as pharmaceutical, food, and beverage. 
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Unfortunately, the term “user” is vague — there are us- 
er account names, computer account names, domain 
names, host names, and of course the human user’s iden- 
tity. While the latter is what is most often required for com- 
pliance management (see Chapter 13, “Standards and 
Regulations”), the former are what are typically provided 
within digital systems. Authentication to a system typically 
requires credentials in the form of a username and pass- 
word, from a machine that has a host name, which might 
be one of several hosts in a named domain. The applica- 
tion itself might then authenticate to another backend sys- 
tem (such as a database), which has its own name and 
to which the application authenticates using yet another 
set of credentials. To further complicate things, the same 
human operator might need to authenticate to several 
systems, from several different machines, and may use 
a unique username on each. As mentioned earlier, ICS 
users may utilize a “common” Windows account shared 
by many, while each possesses a unique “application” ac- 
count used for authentication and authorization within the 
ICS applications. 

It is therefore necessary to normalize users to a com- 
mon identity, just as it is necessary to normalize events 
to a common taxonomy. This can be done by monitoring 
activities from a variety of sources (network, host, and ap- 
plication logs), extracting whatever user identities might 
be present, and correlating them against whatever clues 
might be preset within those logs. For example, if a user 
authenti- cates to a Windows machine, launches an ap- 
plication and authenticates to it, and then the application 
authenticates to a backend system, it is possible to track 
that activity back to the original username by looking at 
the source of the authentications and the time at which 
they occurred. It can be assumed that all three authentica- 
tions were by the same user because they occurred from 
the same physical console in clear succession. 

As the systems become more complex and distributed, and 
as the number of users increases, each with specific roles 
and privileges, this can become cumbersome, and an au- 
tomated identity management mechanism may be required. 

This process is made simpler through the use of com- 
mon directories, such as Microsoft Active Directory and/or 
the Lightweight Directory Access Protocol (LDAP), which 
act as identity directories and repositories. However, there 
may still be several unique sets of credentials per human 
operator that are managed locally within the applications 
versus centrally via a directory service. The difficulty lies 
in the lack of common log formats, and the correspond- 
ing lack of universal identities between diverse systems. 
User monitoring therefore requires the extraction of us- 
er information from a variety of network and application 
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logs, followed by the normalization of that identity informa- 
tion. John Doe might log into a Windows domain using the 
username j.doe, have an e-mail address of j|doe@compa- 
ny.com, and log into a corporate intranet or Content Man- 
agement System (CMS) as johnnyd, and so on. To truly 
monitor user behavior, it is necessary to recognize j.doe, 
jdoe, and johnnyd as a single identity. 

Several commercial identity and access management 
(IAM) systems (also sometimes referred to as identity and 
authentication management systems) are available to fa- 
cilitate this process. Some commercially available IAM 
systems include: NetIQ (formerly Novell and spun off as 
part of the merger with Attachmate), Oracle Identity Man- 
agement (also encompassing legacy Sun Identity Man- 
agement prior to Oracle’s acquisition of Sun Microsys- 
tems), and IBM’s Tivoli Identity. Other third-party identity 
solutions, such as Securonix Identity Matcher, offer fea- 
tures of both a centralized directory and IAM by mining 
identity information from other IAMs and normalizing ev- 
erything back to a common identity." More sophisticated 
SIEM and Log Management systems might also incor- 
porate identity correlation features to provide user nor- 
malization. An authoritative source of identity is provided 
by managing and controlling authentications to multiple 
systems via a centralized IAM irrespective of the method 
used, as shown in Figure 4. 

Once the necessary identity context has been obtained, 
it can be utilized in the information and event manage- 
ment process to cross-reference logs and events back to 
users. A SIEM dashboard shows both network and event 
details associated with their source users in Figure 5. 


11 Securonix, Inc., Securonix Indentity Matcher: Overview. http://www.securonix.com/ 
identity.htm, 2003 (cited: March 3, 2011). 


| Windows login: Username JohnDoe 
Application login: Username jdoe 


| Network login: Username john @domain.com | 


First Name: John 
Last Name: Doe 
Employee ID: #rreeF 
Access Level: Admin 
Zone Authority: 0-4 
Email: 
john.doe Pdomain.co 
m 


PROG: seee-eate-aotee 
e Bte 


| Application backend login: Username commonApp 


Figure 4. Normalization of user identity 
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Figure 5. User activity related to file access as displayed by an SIEM 


Table 2. Contextual Information Sources and Their Relevance 


Directory services (e.g. 
active directory) 


Identity and 
authentication 
management systems 


Vulnerability scanner 


Penetration tester 


Threat database/ CERT 
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User identity information, asset identity information, 
and access privileges 


Detailed user identity information, usernames and 
account aliases, access privileges, and an audit trail of 
authentication activity 


Asset details including the operating system, 
applications in use (ports and services), patch levels, 
identified vulnerabilities, and related known exploits 


Exploitation success/failure, method of exploitation, 
evasion techniques, etc. 


Details, origins and recommendations for the 
remediation of exploits, malware, evasion techniques, 
etc. 

Threat intelligence may also be used as “watchlists,” 
providing a cross-reference against which threats can 
be compared in order to highlight or otherwise call 
out threats of a specific category, severity, etc. 


= | S:\Vapan 
S:\lran 
5:\Jodan 
H:\home 
S:\Indonesia 
H:\ bese 
S:\Ching 
S:\Chile 


H:\wsersl 


H: wsers3 ie 


Provides a repository of known users, assets, and roles 
that can be leveraged for security threat analysis and 
detection, as well as for compliance 


Enables the correlation of users to access and 
activities based upon privilege and policy. When 
used to enrich security events, provides a clear audit 
trail of activity versus authority that is necessary for 
compliance auditing 


Enables security events to be weighted based upon 
the vulnerability of their target (i.e. a Windows virus is 
less concerning if it is targeting a Linux workstation) 


Also provides valuable asset details for use in 
exception reporting, event correlation, and other 
functions 


Like with a vulnerability scanner, pen test tools 
provide the context of an attack vector. Unlike VA 
scan results, which show what could be exploited, 
a pen test indicates what has been exploited — 
which is especially useful for determining evasion 
techniques, detecting mutating code, etc. 


Threat intelligence can be used in a purely advisory 
Capacity (e.g. providing educational data associated 
with a detected threat), or in an analytical capacity 
(e.g. in association with vulnerability scan data to 
weight the severity calculation of a detected threat) 
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ADDITIONAL CONTEXT 

While user identity is one example of contextual informa- 
tion, there is a wealth of additional information available 
that can provide context. This information — such as vul- 
nerability references, IP reputation lists, and threat direc- 
tories — supplements the monitored logs and events with 
additional valuable context. Examples of contextual infor- 
mation are provided in Table 2. 

Contextual information is always beneficial, as the more 
context is available for any specific event or group of 
events, the easier it will be to assess relevance to specific 
security and business policies. This is especially true be- 
cause the logs and events being monitored often lack the 
details that are most relevant, such as usernames (see 
Figure 6).'2 

It is important to Know that contextual information adds 
to the total volume of information already being assessed. 
It is therefore most beneficial when used to enrich other 
security information in an automated manner (see section 
“Information Management”). 


BEHAVIOR 

Behavior is not something that is directly monitored, rath- 
er itis the analysis of any monitored metric (obtained from 
a log, network flow, or other source) over time. The re- 
sult is an indication of expected versus unexpected activ- 
ity, which is extremely useful for a wide range of security 
functions, including anomaly-based threat detection, as 
well as capacity or threshold-based alarming. Behavior is 


12 A. Chuvakin, Content Aware SIEM. http://www.sans.org/security-resources/idfaq/vlan. 
php February, 2000 (cited: January 19, 2011). 


What else happened at this time? 
Near this time? 
What is the Time Zone? 


messages did it produce? 


What is the system's IP 
address? Other names? 
Location in the network? 
Geo location? Who is the 


owner? Who is the 
administrator? What else 
happened to this system? 


Figure 6. A /og file, illustrating the lack of context image 
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' What is this service? What other 


What other systems does it run on? | 


Who is this user? Where is 
he/she coming from? What is 
his/her real name? What 
organizational or business unit? 
Roles? Privileges? 


| What does this number mean? 
| Is this documented somenwere? 


also a useful condition in security event correlation (see 
Chapter 11, “Exception, Anomaly, and Threat Detection”). 

Behavior analysis is often provided by security log and 
event monitoring tools, such as log management systems, 
SIEMs, and network behavior anomaly detection (NBAD) 
systems. If the system used for the collection and moni- 
toring of security information does not provide behavioral 
analysis, an external tool, such as a spreadsheet or sta- 
tistics program, may be required. 


SUCCESSFULLY MONITORING SECURITY ZONES 
Understanding what to monitor is only the first step — actu- 
ally monitoring all of the users, networks, applications, as- 
sets, and other activities still needs to happen. The discus- 
sion of what to monitor focused heavily on logs, because 
log files are designed to describe activities that have oc- 
curred, are fairly ubiquitous, and are well understood. Log 
files are not always available however, and may not pro- 
vide sufficient detail in some instances. Therefore, moni- 
toring is typically performed using a combination of meth- 
ods, including the following: 


¢ Log collection and analysis 
¢ Direct monitoring or network inspection 
¢ Inferred monitoring via tangential systems. 


Except in pure log-collection environments, where logs 
are produced by the assets and network devices that are 
already in place, specialized tools are required to mon- 
itor the various network systems. The results of moni- 


DNS name? Windows name? Other names? Whois info? 
Organization owner? 

Where does the IP originate from (Geo Location)? Who is the 
owner and adminsitrator? 


What else happened on this host? What other hosts did this 
host communicate to? Is there extemal information iavailable 
(Dshield)? 


What is this service? Where 
else does it show up in logs? 


What is this port? What is a 
common service that utilizes this 
port? Where else is this service 
being used? 
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toring (by whatever means) needs to be dealt with, be- 
cause while manual logs and event reviews are possible 
(and allowed by most compliance regulations), automat- 
ed tools are available and are recommended. 

The central analysis of monitored systems is contrary 
to a security model built upon functional isolation. This is 
true because industrial networks should be separated into 
functional security zones, and centralized monitoring re- 
quires that log and event data either remain within a func- 
tional group (limiting the value for overall situation aware- 
ness of the complete system) or be shared between zones 
(potentially putting the security of the zone at risk). In the 
first scenario, logs and events are not allowed across the 
zone perimeter where they may be collected, retained, 
and analyzed only by local systems within that zone. 
In the second scenario, special considerations must be 
made for the transportation of log and event data across 
zone perimeters to prevent the introduction of a new in- 
bound attack vector. A common method is to implement 
special security controls (such as a data diode, unidirec- 
tional gateway, or firewall configured to explicitly deny all 
inbound communications) to ensure that the security data 
are only allowed to flow toward the centralized manage- 
ment system. Ahybrid approach may be used in industrial 
networks where critical systems in remote areas need to 
operate reliably. This provides local security event and log 
collection and management so that the zone can operate 
in total isolation, while also pushing security data to a cen- 
tral location to allow for more complete situational aware- 
ness across multiple zones. 


LOG COLLECTION 

Log collection is simply the collection of logs from what- 
ever sources produce them. This is often a matter of di- 
recting the log output to a log aggregation point, such as 
a network storage facility and/or a dedicated Log Man- 
agement system. Directing a log is often as simple as di- 
recting the syslog event data service to the IP address of 
the aggregator. In some cases, such as WMI, events are 
stored locally within a database rather than as log files. 
These events must be retrieved, either directly (by au- 
thenticating to Windows and querying the event database 
via the Windows Event Collector functionality) or indirectly 
(via a software agent, such as Snare, which retrieves the 
events locally and then transmits them via standard sys- 
log transports). 


DIRECT MONITORING 

Direct monitoring refers to the use of a “probe” or other de- 
vice to passively examine network traffic or hosts by plac- 
ing the device in-line with the network. Direct monitoring is 
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especially useful when the system being monitored does 
not produce logs natively (as is the case with many indus- 
trial network assets, such as RTUs, PLCs, and IEDs). It is 
also useful as a verification of activity reported by logs, 
as log files can be altered deliberately in order to hide 
evidence of malicious activities. Common monitoring de- 
vices include firewalls, intrusion detection systems (IDSs), 
database activity monitors (DAMs), application monitors, 
and network probes. These are often available commer- 
cially as software or appliances, or via open-source dis- 
tributions, such as Snort (IDS/ IPS), Wireshark (network 
sniffer and traffic analyzer), and Kismet (wireless sniffer). 

Often, network monitoring devices produce logs of their 
own, which are then collected for analysis with other logs. 
Network monitoring devices are sometimes referred to as 
“passive logging’ devices because the logs are produced 
without any direct interaction with the system being mon- 
itored. Database activity monitors, for example, monitor 
database activity on the network — often on a span port 
or network tap. The DAM decodes network packets and 
then extracts relevant SQL transactions in order to pro- 
duce logs. There is no need to enable logging on the da- 
tabase itself resulting in no performance impact to the da- 
tabase servers. 

In industrial networks, it is similarly possible to monitor 
industrial protocol use on the network by providing “pas- 
sive logging’ to those industrial control assets that do not 
support logging. Passive monitoring is especially impor- 
tant in these networks, as many industrial protocols op- 
erate in real time and are highly susceptible to network 
latency and jitter. This is one reason why it is difficult to 
deploy logging agents on the devices themselves (which 
would also complicate asset testing policies), making 
passive network logging an ideal solution in these cases. 
Special consideration to any industrial network redundan- 
cy should also be considered when deploying network- 
based monitoring solutions. 

In some instances, the device may use a proprietary log 
format or event streaming protocol that must be handled 
specially. Cisco’s Security Device Event Exchange proto- 
col (SDEE) (used by most Cisco IPS products) requires 
a username and password in order to authenticate with 
the security device so that events can be retrieved on de- 
mand, and/or “pushed” via a subscription model. While 
the end result is the same, it is important to understand 
that syslog is not absolutely ubiquitous. 


INFERRED MONITORING 

Inferred monitoring refers to situations where one system 
is monitored in order to infer information about another 
system. Many applications connect to a database. So as 
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an example, monitoring the database in lieu of the appli- 
cation itself will provide valuable information about how 
the application is being used, even if the application itself 
is not producing logs or being directly monitored by an Ap- 
plication Monitor. 


NOTE 

Network-based monitoring inevitably leads to the ques- 
tion, “Is it possible to monitor encrypted network traffic?” 
Many industrial network regulations and guidelines rec- 
ommend the encryption of control data when these da- 
ta are transferred between trusted security zones via un- 
trusted conduits ... so how can these data be monitored 
via a network probe? There are a few options, each with 
benefits and weaknesses. The first is to monitor the sensi- 
tive network connection between the traffic source and the 
point of encryption. That is, encrypt network traffic exter- 
nally using a network-based encryption appliance, such 
as the Certes Networks Enforcement Point (CEP) vari- 
able speed encryption appliances, and place the network 
probe immediately between the asset and the encryption. 
The second option is to utilize a dedicated network-based 
decryption device, such as the Netronome SSL Inspector. 
These devices perform deliberate, hardware-based man- 
in-themiddle attacks in order to break encryption and ana- 
lyze the network contents for security purposes. A third 
option is not to monitor the encrypted traffic at all, but rath- 
er to monitor for instances of data that should be encrypt- 
ed (such as industrial protocol function codes) but are not 
producing exception alerts indicating that sensitive traffic 
is not being encrypted. 

To determine which tools are needed, start with your 
zone’s perimeter and interior security controls (see Chap- 
ter 9, “Establishing Zones and Conduits”) and determine 
which controls can produce adequate monitoring and 
which cannot. If they can, start by aggregating logs from 
the absolute perimeter (the demarcation between the 
least critical Zone and any untrusted networks — typically 
the business enterprise LAN) to a central log aggregation 
tool (see the section “Information Collection and Manage- 
ment Tools”). Begin aggregating logs from those devices 
protecting the most critical Zones, and work outward un- 
til all available monitoring has been enabled, or until the 
capacity of your log aggregation has become saturated. 
At this point, if there are remaining critical assets that are 
not being effectively monitored, it may be necessary to in- 
crease the capacity of the log aggregation system. 


TIP 


Adding capacity does not always mean buying larger, 
more expensive aggregation devices. Distribution is al- 
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so an option — keep all log aggregation local within each 
zone (or within groups of similar zones), and then aggre- 
gate subsets of each zone to a central aggregation facility 
for centralized log analysis and reporting. While this type 
of event reduction will reduce the effectiveness of threat 
detection and will produce less comprehensive reports 
from the centralized system, all the necessary monitoring 
and log collection will remain intact within the zones them- 
selves, where they can be accessed as needed. 

This concept Is particularly well-suited for industrial net- 
works in that it allows the creation of a local “dashboard” 
where relevant events for nearby assets can be displayed 
and responded to quickly by a “first responder” that may 
reside in the operational or plant environment, while offer- 
ing the ability to export these events to upper-level aggre- 
gators that have a much broader view of more assets, and 
can focus more on event correlation and threat analysis 
typically performed in a security operations center. 


lf all logs are being collected and there are still criti- 
cal assets that are not adequately monitored, it may be 
necessary to add additional network monitoring tools to 
compensate for these deficiencies. This process is illus- 
trated in Figure 7. 
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CAUTION 

Remember that when aggregating logs it is still neces- 
sary to respect the boundaries of all established secu- 
rity zones. If logs need to be aggregated across zones 
(which is helpful for the detection of threats as they move 
between zones), make sure that the zone perimeter is 
configured to only allow the movement of logs in one di- 
rection; otherwise, the perimeter could potentially be com- 
promised. In most instances, simply creating a policy that 
explicitly states the source (the device producing logs) 
and the destination (the log aggregation facility) for the 
specified service (e.g. syslog, port 514) is sufficient in or- 
der to enforce a restricted one-way transmission of the log 
files. For critical zones, physical separation using a data 
diode or unidirectional gateway may be required to assure 
that all log transmissions occur in one direction, and that 
there is no ability for malicious traffic to enter the secure 
zone from the logging facility. 

Additional monitoring tools might include any asset or 
network monitoring device, including host-based security 
agents, or external systems, such as an intrusion detec- 
tion system, an application monitor, or an industrial proto- 
col filter. Network-based monitoring tools are often easier 
to deploy, because they are by nature nonob- trusive and, 
if configured to monitor a spanned or mirrored interface, 
typically do not introduce latency. 


INFORMATION COLLECTIONAND 
MANAGEMENT TOOLS 

The “log collection facility” is typically a log management 
system or a security information and event management 
(SIEM) system. These tools range from very simple to 
very complex and include free, open-source, and com- 
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mercial options. Some options include syslog aggregation 
and log search, commercial log management systems, 
the open source security information management (OS- 
SIM) system, and commercial security information and 
event management systems. 


Syslog Aggregation and Log Search 

Syslog allows log files to be communicated over a net- 
work. By directing all syslog outputs from supported as- 
sets to a common network file system, a very simple and 
free log aggregation system can be established. While in- 
expensive (essentially free), this option provides little add- 
ed value in terms of utilizing the collected logs for anal- 
ysis, requiring the use of additional tools, such as open 
source log search or IT search tools, or through the use 
of a commercial log management system or SIEM. If logs 
are being collected for compliance purposes as well as for 
security monitoring, additional measures will need to be 
taken to comply with log retention requirements. These 
requirements include nonrepudiation and chain of custo- 
dy, as well as ensuring that files have not been altered, 
or accessed by unauthorized users. This can be obtained 
without the help of commercial systems, although it does 
require additional effort by IT managers. 


Log ManagementSystems 

Log management systems provide a commercial solution 
for log collection, analysis, and reporting. Log management 
systems provide a configuration interface to manage log 
collection, as well as options for the storage of logs — of- 
ten allowing the administrator to configure log retention pa- 
rameters by individual log source. At the time of collection, 
log management systems also provide the necessary non- 
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repudiation features to ensure the integrity of the log files, 
such as “signing” logs with a calculated hash that can be 
later compared to the files as a checksum. Once collected, 
the logs can then also be analyzed and searched, with the 
ability to produce prefiltered reports in order to present log 
data relevant to a specific purpose or function, such as com- 
pliance reports, which produce log details specific to one or 
more regulatory compliance controls, as shown in Figure 8. 


Security Information and Event Management 
Systems 

Security information and event management systems, or 
SIEMs, extend the capabilities of log management sys- 
tems with the addition of specific analytical and contextu- 
al functions. According to security analysts from Gartner, 
the differentiating quality of an SIEM is that it combines 
the log management and compliance reporting quali- 
ties of a log management or legacy security information 
management (SIM) system with the real-time monitoring 
and incident management capabilities of a security event 
manager (SEM)."* A SIEM must also support “data cap- 
ture from heterogeneous data sources, including network 
devices, security devices, security programs, 

and servers,’* making the qualifying SIEM an ideal plat- 
form for providing situational awareness across security 
zone perimeters and interiors. 

Many SIEM products are available, including the open- 
source variants (OSSIM by AlienVault), as well as several 
commercial SIEMs (ArcSight by Hewlett-Packard, QRa- 
dar by IBM, LogRhythm, Enterprise Security Manager by 
McAfee, and Splunk Enterprise), competing across a va- 
riety of markets, and offering a variety of valueadded fea- 
tures and specializations. 

Because an SIEM is designed to support real-time mon- 
itoring and analytical functions, it will parse the contents 
of a log file at the time of collection, storing the parsed in- 
formation in some sort of structured data store, typically 
a database or a specialized flat-file storage system. By 
parsing out common values, they are more readily avail- 
able for analytics, helping to support the real-time goals 
of the SIEM, as shown in Figure 9. The parsed data are 
used for analytics, while a more traditional log manage- 
ment framework that will hash the logs and retain them for 
compliance. Because the raw log file may be needed for 
forensic analysis, a logical connection between the log file 
and the parsed event data is typically maintained within 
the data store. 

SIEM platforms are often used in security operations 
centers (SOCs), providing intelligence to security opera- 
13. KM. Kavanagh, M. Nicolet, O. Rochford, “Magic quadrant for security information and 


event management,’ Gartner Document ID Number: G00261641, June 25, 2014. 
14 Ibid. 
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tors that can be used to detect and respond to security 
concerns. Typically, the SIEM will provide visual dash- 
boards to simplify the large amounts of disparate data into 
a more human-readable form. Figure 10 illustrates how a 
custom dashboard is created within Splunk to visual ICS- 
related security events. Figure 11 shows how this dash- 
board can be expanded to provide more application-layer 
event information pertaining to industrial protocol security 
events (e.g. use of invalid function codes). 


NOTE 

Log management and SIEM platforms are converging 
as information security needs become more closely tied 
to regulatory compliance mandates. Many traditional log 
management vendors now offer SIEM features, while tradi- 
tional SIEM vendors are offering log management features. 


Data Historians 

Data Historians are not security monitoring products, but 
they do monitor activity (see Chapter 4, “Introduction to 
Industrial Control Systems and Operations”) and can be 
a useful supplement to security monitoring solutions in 
several ways, including 


¢ Providing visibility into control system assets that may 
not be visible to typical network monitoring tools. 

¢ Providing process efficiency and reliability data that 
can be useful for security analysis. 


Because most security monitoring tools are designed for 
enterprise network use, they are typically restricted to 
TCPand UDP-based IP networks and therefore have no 
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visibility into large portions of most industrial plants that 
may utilize serial connectivity or other nonroutable pro- 
tocols. Many industrial protocols are evolving to oper- 
ate over Ethernet using TCP and UDP transports over 
IP, meaning these processes can be impacted by enter- 
prise network activities. The security analysis capabilities 
of SIEM are made available to operational data by us- 
ing the operational data provided by a Historian, allowing 
threats that originate in IT environments but target OT 
systems (i.e. Stuxnet and Dragonfly) to be more easily 
detected and tracked by security analysts. Those activ- 
ities that could impact the performance and reliability of 
industrial automations systems can be detected as well 
by exposing IT network metrics to operational process- 
es, including network flow activity, heightened latency, or 
other metrics that could impact the proper operation of 
industrial network protocols (see Chapter 6, “Industrial 
Network Protocols”). 


MONITORING ACROSS SECURE BOUNDARIES 

As mentioned in the section “Successfully Monitoring Se- 
curity Zones,” it is sometimes necessary to monitor sys- 
tems across secure zone boundaries via defined con- 
duits. This requires zone perimeter security policies that 
will allow the security logs and events generated by the 
monitoring device(s) to be transferred to a central man- 
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agement console. Data diodes are ideal for this applica- 
tion as they force the information flow in one direction — 
away from the zones possessing higher security levels 
and toward the central management system. If a firewall 
is used, any “hole” provided for logs and events repre- 
sents a potential attack vector. The configuration must 
therefore explicitly limit the communication from the origi- 
nating source(s) to the destination management system, 
by IP (Layer 3), Port (Layer 4), and preferably application 
content (Layer 7), with no allowed return communication 
path. Ideally, this communication would be encrypted as 
well, as the information transmitted could potentially be 
sensitive in nature. 


INFORMATION MANAGEMENT 
The next step in security monitoring is to utilize the rel- 
evant security information that has been collected. Prop- 
er analysis of this information can provide the situational 
awareness necessary to detect incidents that could im- 
pact the safety and reliability of the industrial network. 
Ideally, the SIEM or Log Manager will perform many 
underlying detection functions automatically — includ- 
ing normalization, data enrichment, and correlation (see 
Chapter 11, “Exception, Anomaly, and Threat Detection”) 
— providing the security analyst with the following types of 
information at their disposal: 
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e The raw log and event details obtained by monitoring 
relevant systems and services, normalized to a com- 
mon taxonomy. 

¢ The larger “incidents” or more sophisticated threats 
derived from those raw events that may include corre- 
lation with external global threat intelligence sources. 

¢ The associated necessary context to what has been 
observed (raw events) and derived (correlated events). 


Typically, an SIEM will represent a high-level view of the 
available information on a dashboard or console, as il- 
lustrated in Figure 12, which shows the dashboard of the 
Open Source Security Information Management (OS- 
SIM) platform. With this information in hand, automat- 
ed and manual interaction with the information can oc- 
cur. This information can be queried directly to achieve 
direct answers to explicit questions. It can also be for- 
mulated into a report to satisfy specific business, policy, 
or compliance goals, or it can be used to proactively or 
reactively notify a security or operations officer of an in- 
cident. The information is available to further investigate 
incidents that have already occurred. 


QUERIES 

The term “query” refers to a request for information from 
the centralized data store. This can sometimes be an ac- 
tual database query, using structured query language 
(SQL), or it may be a plain-text request to make the in- 
formation more accessible by users without database ad- 
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ministration skills (although these requests may use SQL 
queries internally, hidden from the user). Common exam- 
ples of initial queries include the following: 


¢ Top 10 talkers (by total network bandwidth used) 
¢ Top talkers (by unique connections or flows) 

¢ Top events (by frequency) 

¢ Top events (by severity) 

¢ Top events over time 

¢ Top applications in use 

¢ Open ports. 


These requests can be made against any or all data that 
are available in the data store (see the section “Data 
Availability”). By providing additional conditions or filters, 
queries can be focused yielding results more relevant to 
a specific situation. For example 


¢ Top 10 talkers during non-business hours 

¢ Top talkers using specific industrial network protocols 

¢ All events of a common type (e.g. user account 
changes) 

¢ All events targeting a specific asset or assets (e.g. 
critical assets within a specific zone) 

¢ All ports and services used by a specific asset or assets 

¢ Top applications in use within more than one zone. 


Query results can be returned in a number of ways: via 
delimited text files, a graphical user interface or dash- 
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Figure 13. An SIEM dashboard showing administrative account changes 
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Figure 14. An exampleof a graphical interface for creating event 
correlation rules 


board, preformatted executive reports, an alert that is 
delivered by SMS or e-mail, and so on. Figure 13 shows 
user activity filtered by a specific event type — in this ex- 
ample, administrative account change activities that cor- 
respond with NERC compliance requirements. 

A defining function of an SIEM is to correlate events to 
find larger incidents (see Chapter 11, “Exception, Anoma- 
ly, and Threat Detection”). This includes the ability to de- 
fine correlation rules, as well as present the results via a 
dashboard. Figure 14 shows a graphical event correlation 
editor that allows the logical conditions (such as “if A and 
B then C”), while Figure 15 shows the result of an incident 
query — in this case the selected incident (an HTTP Com- 
mand and Control Spambot) being derived from four dis- 
crete events. 


REPORTS 

Reports select, organize, and format all relevant data from 
the enriched logs and events into a single document. Re- 
ports provide a useful means to present almost any data 
set. Reports can summarize high-level incidents for exec- 
utives, or include precise and comprehensive documen- 
tation that provides minute details for internal auditing or 
for compliance. An example of a report generated by an 
SIEM is shown in Figure 16 showing a quick summary of 
the OSlsoft Pl Historian authentication failures and point 
change activity. 
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Figure 15. An SIEM dashboard a correlated event and its source events 
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ALERTS 

Alerts are active responses to observed conditions within 
the SIEM. An alert can be a visual notification in a console 
or dashboard, a direct communications (e-mail, page, 
SMS, etc.) to a security administrator, or even the execu- 
tion of a custom script. Common alert mechanisms used 
by commercial SIEMs include the following: 


¢ Visual indicators (e.g. red, orange, yellow, green) 

¢ Direct notification to a user or group of users 

¢ Generation and delivery of a specific report(s) to 
a user or group of users 

¢ Internal logging of alert activity for audit control 

¢ Execution of a custom script or other external control 

¢ Generation of a ticket in a compatible help desk or in- 
cident management system. 


Several compliance regulations, including NERC CIP, 
CFATS, and NRC RG 5.71, require that incidents be ap- 
propriately Communicated to proper authorities inside 
and/or outside of the organization. The alerting mech- 
anism of an SIEM can facilitate this process by creating 
a useable variable or data dictionary with appropriate con- 
tacts within the SIEM and automatically generating appro- 
priate reports and delivering them to key personnel. 


INCIDENT INVESTIGATIONAND RESPONSE 

SIEM and log management systems are useful for incident 
response, because the structure and normalization of the 
data allow an incident response team to drill into a specific 
event to find additional details (often down to the source log 
file contents and/or captured network packets), and to pivot 
on specific data fields to find other related activities. For ex- 
ample, if there is an incident that requires investigation and 
response, it can be examined quickly providing relevant de- 
tails, such as the username and IP address. The SIEM can 
then be queried to determine what other events are associ- 
ated with the user, IP, and so on. In some cases the SIEM 
may support active response capabilities, including 


¢ Allowing direct control over switch or router interfaces 
via SNMP, to disable network interfaces. 

¢ Executing scripts to interact with devices within the 
network infrastructure, to reroute traffic, isolate users, 
and so on. 

¢ Execute scripts to interact with perimeter security de- 
vices (e.g. firewalls) to block subsequent traffic that 
has been discovered to be malicious. 

¢ Execute scripts to interact with directory or IAM sys- 
tems to alter or disable a user account in response to 
observed malicious behavior. 
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Report Generated: Mar 4, 2011 1:58 PM 

Time Zone: Greenwich Mean Time : Dublin, Edinburgh, Lisbon, 
London GMT+00:00 
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Figure 16. An SIEM report showing industrial activities 
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These responses may be supported manually or auto- 
matically, or both. 


CAUTION 

While automated response capabilities can improve ef- 
ficiencies, they should be limited to non-critical security 
zones and/or to zone perimeters. As with any control de- 
ployed within industrial networks, all automated respons- 
es should be carefully considered and tested prior to 
implementation. A false positive could trigger such a re- 
sponse and cause the failure of an industrial operation, 
with potentially serious consequences. 


LOG STORAGEAND RETENTION 

The end result of security monitoring, log collection, and 
enrichment is a large quantity of data in the form of log 
files, which must be stored for audit and compliance pur- 
poses (in the cases where direct monitoring is used in lieu 
of log collection, the monitoring device will still produce 
logs, which must also be retained). This represents a few 
challenges, including how to ensure the integrity of the 
stored files (a common requirement for compliance), how 
and where to store these files, and how they can be kept 
readily available for analysis. 


NONREPUDIATION 

Nonrepudiation refers to the process of ensuring that a log 
file has not been tampered with, so that the original raw 
log file can be presented as evidence, without question of 
authenticity, within a court of law. This can be achieved in 
several ways, including digitally signing log files upon col- 
lection as a checksum, utilizing protected storage media, 
or the use of third-party FIM systems. 

A digital signature is typically provided in the form of 
a hash algorithm that is calculated against the log file at 
the time of collection. The result of this calculation pro- 
vides a checksum against which the files can be verified 
to ensure they have not been tampered with. If the file is 
altered in any way, the hash will calculate a different value 
and the log file will fail the integrity check. If the checksum 
matches, the log is known to be in its original form. 

The use of appropriate storage facilities can ensure 
nonrepudiation as well. For example, by using write once 
read many (WORM) drives, raw log records can be ac- 
cessed but not altered, as the write capability of the drive 
prevents additional saves. Many managed storage area 
network (SAN) systems also provide varying levels of au- 
thentication, encryption, and other safeguards. 

A FIM may already be in use as part of the overall secu- 
rity monitoring infrastructure, as described in the section 
“Assets.” The FIM observes the log storage facility for any 
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sign of changes or alterations, providing an added level of 
integrity validation. 


DATA RETENTION/STORAGE 
The security monitoring tools just mentioned all require 
the collection and storage of security-related information. 
The amount of information that is typically required could 
easily surpass 170 GB over an 8-h period for a medium- 
sized enterprise collecting information at approximately 
20,000 events per second.'® It is worth mentioning that 
event generation within an industrial network is typically 
a small fraction of this number, and when properly tuned, 
presents a manageable amount of information storage. 
Data retention refers to the amount of information that 
is stored long-term, and can be measured in volume (the 
size of the total collected logs in bytes) and time (the num- 
ber of months or years that logs are stored for). The length 
of time a log is retained is important, as this metric is often 
defined by compliance regulations — NERC CIP requires 
that logs are retained for anywhere from 90 days to up to 3 
years, depending upon the nature of the log.'* The amount 
of physical storage space that is required can be calculated 
by determining which logs are needed for compliance and 
for how long they must be kept. Some of the factors that 
should be considered include the following: 


¢ Identifying the quantity of inbound logs 

¢ Determining the average log file size 

¢ Determining the period of retention required for logs 

¢ Determining the supported file compression ratios of 
the log management or SIEM platform being used. 


Table 3 illustrates how sustained log collection rates map 
to total log storage requirements over a retention period of 
7 years, resulting in a few terabytes (10'7) of storage up to 
hundreds of terabytes or even petabytes (10'°) of storage. 

There may be a requirement to retain an audit trail for 
more than one standard or regulation depending upon 
the nature of the organization, often with each regula- 
tion mandating different retention requirements. As with 
NERC CIP, there may also be a change in the retention 
requirements depending upon the nature of the log, and 
whether an incident has occurred. All of this adds up to 
even greater, long-term storage requirements. 


TIP 

Make sure that the amount of available storage has suf- 
ficient headroom to accommodate spikes in event activity, 
15 J.M. Butler, Benchmarking Security Information Event Management (SIEM). The SANS Institute Analytics 
Program, February, 2009. 


16 North American Electric Reliability Corporation. NERC CIP Reliability Standards, version 4. http://www.nerc. 
com/page.php?cid=2|20 February 3, 2011 (cited: March 3, 2011). 


03/2015 


because event rates can vary (especially during a security 
incident). 


DATA AVAILABILITY 

Data availability differs from retention, referring to the 
amount of data that is accessible for analysis. Also called 
‘live’ or “online” data, the total data availability deter- 
mines how much information can be analyzed concur- 
rently — again, in either volume (bytes and/or total num- 
ber of events) or time. Data retention affects the ability 
of an SIEM to detect “low and slow” attacks (attacks that 
purposefully occur over a long period of time in order to 
evade detection), as well as to perform trend analysis and 
anomaly detection (which by definition requires a series 
of data over time — see Chapter 11, “Exception, Anomaly, 
and Threat Detection’). 


TIP 

In order to meet compliance standards, it may be neces- 
sary to produce a list of all network flows within a particular 
security zone that originated from outside of that zone, for 
the past 3 years. For this query to be successful, 3 years 
of network flow data need to be available to the SIEM at 
once. There is a work-around if the SIEM’s data availabil- 
ity is insufficient (for example, it can only keep 1 year of 
data active). The information can be stored in volumes 
consistent with the SIEM’s data availability by archiving 
older data sets. A partial result is obtained by querying the 
active data set. Two additional queries can be run by then 
restoring the next-previous backup or archive, producing 
multiple partial result sets of 1 year each. These results 
can then be combined to obtain the required 3-year re- 
port. Note that this requires extra effort on the part of the 
analyst. The archive/retrieval process on some legacy 
SIEMs may interfere with or interrupt the collection of new 
logs until the process is complete. 


Table 3. Log Storage Requirements Over Time 


100,000 8.64 3154 508 
50,000 4.32 1577 508 
25,000 2.16 788 508 
10,000 0.86 315 508 
5,000 0.43 158 508 
1,000 0.09 32 508 
500 0.04 16 508 
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Unlike data retention, which is bound by the available 
volume of data storage (disk drive space), data availabil- 
ity is dependent upon the structured data that are used 
by the SIEM for analysis. Depending upon the nature of 
the data store, the total data availability of the system 
may be limited to a number of days, months, or years. 
Typically, one or more of the following limits databases: 


¢ The total number of columns (indices or fields) 

¢ The total number of rows (discreet records or events) 

¢ The rate at which new information is inserted (i.e. col- 
lection rate) 

¢ The rate at which query results are required (i.e. re- 
trieval rates). 


Depending upon the business and security drivers be- 
hind information security monitoring, it may be neces- 
sary to segment or distribute monitoring and analysis in- 
to zones to meet performance requirements. Some fac- 
tors to consider when calculating the necessary data 
availability include 


¢ The total length of time over which data analysis may 
be required by compliance standards. 

¢ The estimated quantity of logs that may be collected 
in that time based on event estimates. 

¢ The incident response requirements of the organiza- 
tion — certain governmental or other critical installa- 
tions may require rapid-response initiatives that ne- 
cessitate fast data retrieval. 

¢ The desired granularity of the information that is kept 
available for analysis (i.e. are there many vs. few indices). 


SUMMARY 

A larger picture of security-related activity begins to form 
once zone security measures are in place. Exceptions 
from the established security policies can then be detect- 
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ed by measuring these activities and further analyzing 
them. Anomalous activities can also be identified so that 
they may be further investigated. 

This requires well-defined policies with those policies 
configured within an appropriate information analysis tool. 
Just as with perimeter defenses to the security zone, care- 
fully built variables defining allowed assets, users, appli- 
cations, and behaviors can be used to aid in detection 
of security risks and threats. If these lists can be deter- 
mined dynamically, in response to observed activity with- 
in the network, the “whitelisting” of known-good policies, 
becomes “smart-listing.” This helps further strengthen pe- 
rimeter defenses through dynamic firewall configuration 
or IPS rule creation. 

The event information can be further analyzed as vari- 
ous threat detection techniques are used together by 
event correlation systems that find larger patterns more 
indicative of serious threats or incidents. Widely used in IT 
network security, event correlation is beginning to “cross 
the divide” into OT networks, at the heels of Stuxnet and 
other sophisticated threats that attempt to compromise 
industrial network systems via attached IT networks and 
services. 

Everything (measured metrics, baseline analysis, and 
whitelists) rely on a rich base of relevant security infor- 
mation. Where does this security information come from? 
The networks, assets, hosts, applications, protocols, us- 
ers, and everything else that is logged or monitored con- 
tributes to the necessary base of data required to achieve 
“situational awareness” and effectively secure an indus- 
trial network. 


Eric D. Knapp is a recognized expert in industrial control systems 
(ICS) cyber security. He is the original author of “Industrial Network 
Security: Securing Critical Infrastructure Networks for Smart Grid, 
SCADA, and Other Industrial Control Systems (First Edition)” and the 
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coauthor of “Applied Cyber Security for Smart Grids.” Eric has held 
senior technology positions at NitroSecurity, McAfee, Wurldtech, and 
Honeywell, where he has consistently focused on the advancement 
of end-to-end ICS cyber security in order to promote safer and more 
reliable automation infrastructures. Eric has over 20 years of expe- 
rience in Information Technology, specializing in cyber security an- 
alytics, threat, and risk management techniques and applied Ether- 
net protocols in both enterprise and industrial networks. 

In addition to his work in information security, Eric is an award-win- 
ning fiction author. He studied English and Writing at the University 
of New Hampshire and the University of London, and holds a degree 


in communications. 


Joel Thomas Langill brings a unique perspective to operational se- 
curity with decades of experience in industrial automation and con- 
trol. He has deployed ICS solutions covering most major industry 
sectors globally encompassing most generations of automated con- 
trol. He has been directly involved in automation solutions span- 
ning feasibility, budgeting, front-end engineering design, detailed 
design, system integration, commissioning, support and legacy sys- 
tem migration. Joel is currently an independent consultant provid- 
ing services to ICS suppliers, end-users, system integrators, and gov- 
ernmental agencies worldwide. Joel founded the popular ICS secu- 
rity website SCADAhacker.com offering visitors resources in under- 
standing, evaluating, and securing control systems. He developed a 
specialized training curriculum that focuses on applied cyber secu- 
rity and defenses for industrial systems. His website and social net- 
works extends to readers in over 100 countries globally. 

Joel serves on the Board of Advisors for Scada Fence Ltd., and is an 
ICS research focal point to corporations and CERT organizations 
around the world. He is a voting member of the ISA99 committee, 
and has published numerous reports on ICS-related campaigns in- 
cluding Heartbleed, Dragonfly, and Black Energy. He is a graduate of 
the University of Illinois-Champaign with a BS (University Honors/ 
Bronze Tablet) in Electrical Engineering. 

He can be found on Twitter @SCADAhacker 
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With a former intelligence operative 
confirming that the NSA has developed 

the prized technique of concealing spyware 

in the firmware of hard drives, what are the 
implications and Is there any point in shutting 
the door now that the horse has bolted? 


y wife, bless her, has to put up with a demanding 
\/ home chef who in her own words, is “a good cook 

but a messy cook”. To further add insult to inju- 
ry, on a trip to the supermarket | will religiously read the 
ingredients of every packet | purchase, unless of course 
| am confident of the brand, but even then | still have my 
suspicions. The age old traders’ trick of placing a thumb 
on the weighing scales or selling adulterated produce did 
not die out in the Victorian age, despite all the legislation 
and government control we live under in the 21st century. 
What concerns me most is provenance, where the prod- 
uct was produced and its roots. For instance, fake saf- 
fron is a lucrative business when the genuine article can 
sell upwards of $7000 per kilogram, approximately tenfold 
that of raw Colombian cocaine prior to processing. As al- 
ways, the Latin phrase Caveat emptor springs to mind — 
buyer beware. 

For too long like the banking sector, the IT industry has 
based physical and electronic transactions on the basis of 
trust. Trust is the fundamental pillar backed by commercial 
law, and once that trust has eroded, only paranoia, panic 
and protectionism can seek to redress the balance. May- 
be it is just me, but seasoned IT professionals tend to lean 
towards the paranoid, as technologists having grasped 
the hinterland of J. Robert Oppenheimer’s quote from the 
Bhagavad Gita — "Now | am become Death, the destroyer 
of worlds”. Realizing with great power comes responsi- 
bility, we lean towards the conservative, not wanting to 
take unnecessary risks, yet at the same time powerless 
in the face of a commercial and political juggernaut that 
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abandons value and is ignorant of the nuances of the dan- 
gers when technology becomes the master rather than 
the slave. We are all but cogs in the machine. 


lf the Reuters report’ is accurate, and there is no reason 
technologically or conspiratorially to believe otherwise, 
we have crossed the Rubicon — reached a point of no re- 
turn. Pandora’s box has been opened, and the IT indus- 
try needs to grasp the implications of this revelation. It is 
not so much that systems can be compromised. We know 
this as fact. What is so disturbing is that the manufacturing 
and commissioning process has been compromised, and 
unlike the defense or airline sectors, we do not have the 
skirts of an official secrets act or American equivalent to 
hide behind. You can wrap an incident in bureaucratic red 
tape and prevent physical access to an aircraft, but you 
cannot stop a horde of Open Source gurus examining kit 
with a hex editor or a decent oscilloscope and a voltmeter. 
Any Fifth Column will have a hard time remaining hidden, 
exponentially leading to the rapid erosion of trust. 


What troubles me is that | have personal experience of 
this, and | can confirm that according to rumor, a major 
European airline in the early 1990’s replaced their flight 
control computers across the entire fleet when they dis- 
covered they were compromised by a foreign power, giv- 
ing them the ability to remotely commandeer an aircraft. 
This was well known in the industry at the time, and | on- 
ly came across this vertical knowledge as | was party to 
installing hardware and software for airlines worldwide 
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at the time. While | am not alone in blowing this particu- 
lar whistle, to date the mainstream media and commer- 
cial culture would prefer that we dismiss this as fantasy 
or conspiracy and anyone subscribing to this as a false 
prophet or suffering from a Cassandra complex. The geo- 
political and ethical implications are obvious post 9/11, yet 
the silence on this incident is deafening. All | can do is at- 
test to what | heard and leave the reader to judge. 


lf we are to accept the fact that physical systems have 
been compromised at the manufacturing level, we must 
seriously consider that potentially even the Open Source 
movement itself has been compromised, either by acci- 
dent or design. The Annus horribilis of the BASH scripting 
bug and SSL compromise will attest to this. While these 
security incidents have been confirmed as the result of hu- 
man error, there are a raft of “Unknown Unknowns” |urk- 
ing out there. To quote political rhetoric, the terrorists only 
need to be lucky once, the security services need to be 
vigilant continually. The rabbit hole extends very deep in- 
deed and the implications are troubling. It all comes down 
to ethics, money and power, and no matter how innocent 
and pure our personal motives, there will always be those 
whose real intention is to pervert, corrupt and compromise 
while appearing as an angel of light. And this goes down 
through the chain, from hardware, firmware and software 
to design, commissioning and management. Statistically, 
there have to be some bad apples in the barrel. 


With a former intelligence... 


We need to grow up as individuals, and as a commu- 
nity realize our value to the world. When the scandal of 
melamine in baby formula in China was exposed, there 
was the usual shock and horror but very little was done 
globally to improve food security and quality. In the West- 
ern world, we delegate this responsibility to our govern- 
ment, and the government points to the law and leaves 
the sector to police itself. Plus ca change, plus mest la 
méme chose. Technology is as critical as food for without 
it, our society — and indeed our civilization — would be in 
dire peril. The current Zeitgeist that looks to reform the es- 
tablished system, be it political, legal or financial is knock- 
ing at our door, and it would be a foolish man or woman 
that attempts to use the excuse of “We are different” as 
a rebuttal to the forces of change or self examination. 
References: 


1. http://www.reuters.com/article/2015/02/16/us-usa-cyberspying- 
idUSKBNOLK1QV20150216 


Rob Somerville has been passionate about technology since his ear- 


ly teens. A keen advocate of open systems since the mid-eighties, he 
has worked in many corporate sectors including finance, automo- 
tive, airlines, government and media in a variety of roles from tech- 
nical support, system administrator, developer, systems integrator 
and IT manager. He has moved on from CP/M and nixie tubes but 
keeps a soldering iron handy just in case. 
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Solene Rapenne 


Q: Could you please introduce yourself and 
your employer? 

A: My name is Soléne Rapenne, I’m a 25 year old IT girl. 
| work for a French company named Carte Blanche Con- 
seil as the (only) IT administrator. Most of our servers run 
FreeBSD and they all use almost exclusively open source 
software. 

Carte Blanche Conseil is a small company founded in 
1989. The company is split into two activities: software 
development and consulting. Our business sector is 
mobility and traffic. We also develop the product MacMap 
which is a user-friendly GIS software for MacOS that we 
may release an open source version of in 2015. 


Q: How did you get started with the BSD 
generation of Operating Systems and when? 
A: | think | tried BSD for the first time with FreeBSD ver- 
sion 6. | was very curious; at this time | was trying a few 
brands of Linux distributions and | wanted to try some- 
thing a bit different, so | chose FreeBSD. | wasn’t very 
familiar with Unix; even if | had used Linux before. | really 
had no idea of what | was doing sometimes. When try- 
ing to tweak the system, | often needed to reinstall from 
scratch because the system was broken and | didn’t know 
how to repair it! 

| learned a lot myself while | was at the university, and 
| have been using both FreeBSD and OpenBSD every- 
day, one on my workstation and the other on my laptop. 
In 2010, at the end of my studies, | successfully passed 
the BSD Associate certification. 


A: What is your favorite BSD OS and can you 
explain what makes it special to you compared 
to the others? 

A: | can't say | have a favorite BSD. | really like FreeBSD, 
OpenBSD and DragonFly BSD. FreeBSD has great per- 
formance and it is stable, with nice features. Meanwhile, 
OpenBSD is stable, secure, very well documented and 
easy to use, but it lacks performance in my opinion. | like 
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playing with FreeBSD, everything in ports can be config- 
ured to use some specific options, system upgrades just 
works like a charm and the new package management is 
awesome. About DragonFly BSD, | like how it’s always be- 
ing improved. The system has gone far in a few years and 
tries to innovate while keeping the system pretty stable. 


Q: What is your approach with the Open Source 
culture and what do you think about it in the 
modern software life-cycle? 

A: | really love Open Source. I’m not sure | would do the 
job | have actually if everything was closed-source. Open 
Source is a great thing where everyone can improve the 
tools for other people. I’m trying to use exclusively open- 
source software when it is possible. | would like to con- 
tribute more to software | like, but very often the projects 
need developers while | can only submit bug reports, 
ideas or some basic patches. 

In France, there is more and more open source soft- 
ware paid for by the French Administration to suit their 
needs. They chose this model because once the con- 
tract is finished, the Administration can keep the sources, 
can receive contributions from volunteers and the proj- 
ect can evolve. If needed, they would pay developers to 
carry out further improvements. In the past, they were 
paying for software, and once the product was delivered, 
they couldn't improve it or fix anything without the seller’s 
contribution. Then, when they really need the software to 
be improved, because of a new law for example, if the 
seller's company doesn’t provide support for the product, 
they have to buy a new product and train people to use it. 


Q: Can you please introduce the ownCloud 
project? 

A: ownCloud is a project that aims to provide to anyone 
the ability to have its own Cloud service and to sync files. 
Part of it is written in PHP for the web interface and back- 
end, and another part for the syncing desktop client (Win- 
dows/Linux/Mac). 
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ownCloud comes by default with a few plugins to al- 
low you to encrypt your files, manage users and set quo- 
tas, picture viewer, calendars, contacts, write documents 
(by using a LibreOffice headless server). You can install 
3rd party applications to add video viewer support, round- 
cube mail integration and a lot more! 

ownCloud is a great tool which allows you to share your 
files as you want. You can share a file or a folder with 
a link that can be set to expire at a given date and also set 
a password for accessing the files. Of course, files can be 
shared between users, and even between different own- 
Cloud installations! 


Q: Are you actively involved in the 
development/management/promotion of the 
ownCloud project? If not, would you like to 
contribute and how? 

A: | am not actually involved in ownCloud. | am promoting 
it “passively” in communities | am in by telling them that 
I'm using it, that it is a nice product and by helping people 
to install and configure it. 


Q: What do you think makes ownCloud 
different from other cloud based storage? 

A: The most important thing with ownCloud is the word 
“own” in its name. It is open-source, you install iton YOUR 
server and do whatever you want with it. You can also 
write plugins for ownCloud or download “apps” made by 
the community. | have never used any other Cloud plat- 
form since | don’t want to share my data with companies. 


Q: Assuming a user has her own data on 
another cloud platform, how easy is it to 
migrate to ownCloud, sync the devices and 
start using this platform? 

ownCloud installation is easy, and it only requires little 
knowledge of UNIX. You can find a lot of tutorials about its 
installation process and on how to configure it. As far | as 
know, ownCloud doesn’t provide any migration tools to re- 
trieve data from another cloud storage provider. 


Q: What makes the special connection between 

DragonFly BSD and ownCloud? 

A: DragonFly BSD is a very nice system, very lightweight, 
ithas good performance and a really interesting filesystem. 
The filesystem HAMMER is really interesting when used 
with a storage utility like ownCloud. The snapshot system 
makes the software upgrade a lot easier when it comes to 
backup everything, because it allows to revert easily some 
files and it doesn’t take any time. It’s also possible to use 
the PFS streaming to replicate the ownCloud data in anoth- 
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er server for high availability. Deduplication can save some 
disk space depending on the kind of data stored. Com- 
pared to ZFS which provides most of these functionalities, 
HAMMER can run on more modest hardware. 


Q: Are there any specific ownCloud events such 

as conferences and meetings? 

A: Yes, ownCloud organizes events all around the world: 
https:/ownCloud.com/events/event/Personally | never 
had the chance to be part of an event like this. 


Q: Do you believe that ownCloud is better 
suited for personal usage or even for enterprise 
adoption? Can you provide some notable 
examples of usage? 
A: For personal use, ownCloud can be more expensive 
than other well-known cloud storage services. Why? You 
need a server, or at least a company selling an ownCloud 
service, while the other cloud services offer you a bunch 
of free Gb. If your data amount fits into a free offer, own- 
Cloud will be more expensive. However, if you have 
a huge amount of data, the ownCloud price will remain 
with a set price, while other services will cost more ac- 
cording to the amount stored. But with ownCloud, you can 
share the charges with other interested people, and even 
set quotas to be sure there will be space for everyone. 
For enterprise adoption, it is different. Very often, com- 
panies already have a tool to share documents between 
employees like a NAS. ownCloud can be used to store, 
share and sync documents but it needs to change hab- 
its compared to a “traditional Windows share’. What own- 
Cloud can really provide to an enterprise is the ability to 
share documents over the Internet. Since a few years, 
| see more and more people from other companies send- 
ing me links to a cloud storage to download some heavy 
files that can’t be sent by mail, but they use a 3rd party 
cloud service, certainly without agreement of their IT ser- 
vice but they don't have any other choice if they want to 
share a file. ownCloud can bring the power to share files 
while keeping control over them. 


Q: Is there a training program to let the users 
start using the platform and/or ease the jump- 
in of enterprises? 

A: ownCloud has an online demo if you want to try the 
product: https://demo.ownCloud.org/. 

There is no training on the product's use, but sometimes 
there are events about (how to install, configure and main- 
tain your ownCloud). The product is pretty easy to install 
anyway, and there is no need to train someone on its use. 
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“Get some sleep beforehand, 
and divide and conquer the packed 


schedule with colleagues.” 
—Paul Reed, Technology Strategy & Innovation, FIS 
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for professionals implementing Big Data solutions 
at their company 


Come to Big Data TechCon to learn the best ways to: 


e Process and analyze the real-time data pouring into your organization 


“Worthwhile, technical, sii a breath of 


fresh air.” 
—Julian Gottesman, ClO, DRA Imaging 


Learn how to extract better data analytics and predictive analysis 
to produce the kind of actionable information and reports your 
organization needs. 


Come up to speed on the latest Big Data technologies like Yarn, Hadoop, 
Apache Spark and Cascading 


Understand HOW to leverage Big Data to help your organization today 


“Big Data TechCon Is definitely worth the 
investment.” 


— Sunil Epari, Solutions Architect, Epari Inc. 


Big Data TechCon” is a trademark of BZ Media LLC. A Event 


Using FreeBSD as a File Server with ZFS 


Ivan Voras 


The ZFS storage workshop will teach you how to create a ZFS file system from scratch and build a file server on top 
of it, but it will also teach you how ZFS, file systems and storage servers work in general. You will learn what ZFS 
looks like, its many features and quirks, and how to use it in a FreeBSD server as a building block of a small file 
server. 


ZFS is the ground-breaking file system originally developed at Sun Inc. for their Solaris operating system. It was 
open-sourced as a part of their OpenSolaris initiative and from there has spread to multiple other operating systems. 
FreeBSD was the first one to implement a working port, and though it has taken a fairly long time of tweaking and 
stabilization, it is now a robust and popular choice. There are products which successfully build upon the technolo- 
gies of FreeBSD and ZFS, such as FreeNAS and its related enterprise-class products from iXsystems, which au- 
tomate and simplify a lot of the tasks, but all of them use the same ZFS interface under the hood, which is not that 
complicated in itself. 


The requirements for this workshop are decent knowledge of FreeBSD, a basic familiarity with command-line op- 
erations, and a system (possibly a virtual machine) on which the student will perform the required tasks, containing 
at least four hard drives (physical or virtual). Since the topic of this workshop is file servers, the participants must 
prepare a virtual or a physical machine with at least two disk drives (and preferably 4), which which to perform the 
exercises and the setup from the workshop. 


http://osdmag.org/course/using-freebsd-as-a-file-server-with-zfs-2/ 


Ivan Voras is a FreeBSD developer and a long-time user, starting with FreeBSD 4.3 and throughout all the versions since. 
In real life he is a researcher, system administrator and a developer, as opportunity presents itself, with a wide range of 
experience from hardware hacking to cloud computing. He is currently employed at the University of Zagreb Faculty of 
Electrical Engineering and Eomputing and lives in Zagreb, Croatia. You can follow him on his blog in English at http:// 
ivoras.net/blog or in Croatian at http://hrblog.ivoras.net/, as well as Google+ at https://plus.google.com/+IvanVoras. 
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